[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260806 04:05:41.538246 3041 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.248.126:34201
I20260806 04:05:41.539847 3041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:05:41.541016 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:05:41.562157 3053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:05:41.562206 3051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:05:41.562165 3050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:05:41.563678 3041 server_base.cc:1034] running on GCE node
I20260806 04:05:42.782982 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:05:42.783262 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:05:42.783540 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989142783498 us; error 0 us; skew 500 ppm
I20260806 04:05:42.784443 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:05:42.796612 3041 webserver.cc:460] Webserver started at http://127.2.248.126:34951/ using document root <none> and password file <none>
I20260806 04:05:42.797642 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:05:42.797904 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:05:42.798408 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:05:42.805140 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/master-0-root/instance:
uuid: "6c3a31e75e6349c0844e2b3d6369efc1"
format_stamp: "Formatted at 2026-08-06 04:05:42 on dist-test-slave-3kk6"
I20260806 04:05:42.813644 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260806 04:05:42.819409 3061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:42.820683 3041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 04:05:42.821027 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/master-0-root
uuid: "6c3a31e75e6349c0844e2b3d6369efc1"
format_stamp: "Formatted at 2026-08-06 04:05:42 on dist-test-slave-3kk6"
I20260806 04:05:42.821362 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:05:42.864861 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:05:42.866453 3041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:05:42.866945 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:05:42.944672 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.126:34201
I20260806 04:05:42.944775 3139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.126:34201 every 8 connection(s)
I20260806 04:05:42.952229 3140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:42.977229 3140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1: Bootstrap starting.
I20260806 04:05:42.983349 3140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:42.985335 3140 log.cc:824] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:05:42.990185 3140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1: No bootstrap required, opened a new log
I20260806 04:05:43.013696 3140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" member_type: VOTER }
I20260806 04:05:43.014379 3140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:43.014709 3140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3a31e75e6349c0844e2b3d6369efc1, State: Initialized, Role: FOLLOWER
I20260806 04:05:43.015583 3140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" member_type: VOTER }
I20260806 04:05:43.016088 3140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:05:43.016340 3140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:05:43.016623 3140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:43.020746 3140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" member_type: VOTER }
I20260806 04:05:43.021577 3140 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c3a31e75e6349c0844e2b3d6369efc1; no voters:
I20260806 04:05:43.023489 3140 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:05:43.023878 3144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:43.026871 3144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3a31e75e6349c0844e2b3d6369efc1, State: Running, Role: LEADER
I20260806 04:05:43.027915 3144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" member_type: VOTER }
I20260806 04:05:43.028998 3140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:05:43.040217 3146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3a31e75e6349c0844e2b3d6369efc1. Latest consensus state: current_term: 1 leader_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" member_type: VOTER } }
I20260806 04:05:43.041410 3146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [sys.catalog]: This master's current role is: LEADER
I20260806 04:05:43.043138 3145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3a31e75e6349c0844e2b3d6369efc1" member_type: VOTER } }
I20260806 04:05:43.044620 3145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [sys.catalog]: This master's current role is: LEADER
I20260806 04:05:43.045713 3163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:05:43.059800 3163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:05:43.061853 3041 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 04:05:43.077703 3163 catalog_manager.cc:1347] Generated new cluster ID: a6317d1935794a59bc32a209af55d46a
I20260806 04:05:43.078004 3163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:05:43.116493 3163 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:05:43.118505 3163 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:05:43.132182 3163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1: Generated new TSK 0
I20260806 04:05:43.133302 3163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:05:43.193990 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:05:43.204540 3179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:05:43.205663 3180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:05:43.211048 3182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:05:43.211994 3041 server_base.cc:1034] running on GCE node
I20260806 04:05:43.212884 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:05:43.213105 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:05:43.213310 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989143213294 us; error 0 us; skew 500 ppm
I20260806 04:05:43.213903 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:05:43.218901 3041 webserver.cc:460] Webserver started at http://127.2.248.65:43529/ using document root <none> and password file <none>
I20260806 04:05:43.219516 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:05:43.219750 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:05:43.220067 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:05:43.221500 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-0-root/instance:
uuid: "2d3a226c500b4528a2dea7cd30910d6a"
format_stamp: "Formatted at 2026-08-06 04:05:43 on dist-test-slave-3kk6"
I20260806 04:05:43.228018 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260806 04:05:43.233350 3188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.234339 3041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 04:05:43.234702 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-0-root
uuid: "2d3a226c500b4528a2dea7cd30910d6a"
format_stamp: "Formatted at 2026-08-06 04:05:43 on dist-test-slave-3kk6"
I20260806 04:05:43.235015 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:05:43.256565 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:05:43.257972 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:05:43.261024 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:05:43.266152 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:05:43.266392 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.266764 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:05:43.266925 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.382123 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.65:46817
I20260806 04:05:43.382221 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.65:46817 every 8 connection(s)
I20260806 04:05:43.389812 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:05:43.405318 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:05:43.411643 3041 server_base.cc:1034] running on GCE node
W20260806 04:05:43.413196 3280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:05:43.414815 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:05:43.416007 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:05:43.416319 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:05:43.416460 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989143416449 us; error 0 us; skew 500 ppm
I20260806 04:05:43.417300 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:05:43.422475 3041 webserver.cc:460] Webserver started at http://127.2.248.66:44049/ using document root <none> and password file <none>
I20260806 04:05:43.423309 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:05:43.423588 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:05:43.423949 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:05:43.425278 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-1-root/instance:
uuid: "9040ee4726534f72b52b4c4fd0188fc3"
format_stamp: "Formatted at 2026-08-06 04:05:43 on dist-test-slave-3kk6"
I20260806 04:05:43.425696 3274 heartbeater.cc:347] Connected to a master server at 127.2.248.126:34201
I20260806 04:05:43.426199 3274 heartbeater.cc:464] Registering TS with master...
I20260806 04:05:43.427378 3274 heartbeater.cc:511] Master 127.2.248.126:34201 requested a full tablet report, sending...
I20260806 04:05:43.431391 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 04:05:43.431398 3087 ts_manager.cc:194] Registered new tserver with Master: 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:05:43.435074 3087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53680
I20260806 04:05:43.436159 3297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.437958 3041 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 04:05:43.438324 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-1-root
uuid: "9040ee4726534f72b52b4c4fd0188fc3"
format_stamp: "Formatted at 2026-08-06 04:05:43 on dist-test-slave-3kk6"
I20260806 04:05:43.438737 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:05:43.455549 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:05:43.456853 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:05:43.459226 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:05:43.462209 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:05:43.462450 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.462759 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:05:43.462939 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.561822 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.66:45521
I20260806 04:05:43.562013 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.66:45521 every 8 connection(s)
I20260806 04:05:43.569007 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:05:43.577883 3387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:05:43.579527 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:05:43.586889 3384 heartbeater.cc:347] Connected to a master server at 127.2.248.126:34201
I20260806 04:05:43.587347 3384 heartbeater.cc:464] Registering TS with master...
I20260806 04:05:43.587456 3041 server_base.cc:1034] running on GCE node
I20260806 04:05:43.588235 3384 heartbeater.cc:511] Master 127.2.248.126:34201 requested a full tablet report, sending...
W20260806 04:05:43.590523 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:05:43.590587 3087 ts_manager.cc:194] Registered new tserver with Master: 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:43.591810 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:05:43.592079 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:05:43.592294 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989143592279 us; error 0 us; skew 500 ppm
I20260806 04:05:43.592530 3087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53694
I20260806 04:05:43.593050 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:05:43.596491 3041 webserver.cc:460] Webserver started at http://127.2.248.67:33789/ using document root <none> and password file <none>
I20260806 04:05:43.597170 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:05:43.597419 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:05:43.597770 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:05:43.599258 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-2-root/instance:
uuid: "8983098d2f3843199e31fe0c278e12a3"
format_stamp: "Formatted at 2026-08-06 04:05:43 on dist-test-slave-3kk6"
I20260806 04:05:43.605698 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 04:05:43.610289 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.611269 3041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:05:43.611636 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-2-root
uuid: "8983098d2f3843199e31fe0c278e12a3"
format_stamp: "Formatted at 2026-08-06 04:05:43 on dist-test-slave-3kk6"
I20260806 04:05:43.612033 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785989141369922-3041-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:05:43.651861 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:05:43.653147 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:05:43.654568 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:05:43.656863 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:05:43.657114 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.657375 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:05:43.657554 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:05:43.762486 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.67:42789
I20260806 04:05:43.763191 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.67:42789 every 8 connection(s)
I20260806 04:05:43.783232 3495 heartbeater.cc:347] Connected to a master server at 127.2.248.126:34201
I20260806 04:05:43.783720 3495 heartbeater.cc:464] Registering TS with master...
I20260806 04:05:43.784675 3495 heartbeater.cc:511] Master 127.2.248.126:34201 requested a full tablet report, sending...
I20260806 04:05:43.786791 3087 ts_manager.cc:194] Registered new tserver with Master: 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:43.787036 3041 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015880554s
I20260806 04:05:43.788483 3087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53700
I20260806 04:05:43.791307 3041 test_util.cc:268] Using random seed: 8193709
I20260806 04:05:43.868170 3087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53714:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:05:43.876196 3087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260806 04:05:44.047219 3162 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:05:44.052320 3441 tablet_service.cc:1467] Processing CreateTablet for tablet 9323adac1819491f8d688196ca1a696a (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 04:05:44.054982 3443 tablet_service.cc:1467] Processing CreateTablet for tablet 749844bfd19e4bdb8bd8aad14b132a78 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 04:05:44.059815 3437 tablet_service.cc:1467] Processing CreateTablet for tablet 18319d34a9154a27946dda911923380a (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 04:05:44.061586 3441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9323adac1819491f8d688196ca1a696a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.063035 3434 tablet_service.cc:1467] Processing CreateTablet for tablet e40f9b110c2e4169b055a1c0de83d19f (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 04:05:44.064291 3435 tablet_service.cc:1467] Processing CreateTablet for tablet dd58b8e6c32b4b1ba07225530f879797 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 04:05:44.063112 3439 tablet_service.cc:1467] Processing CreateTablet for tablet 6b09fbea2f484bf28173d95655b8a0a3 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 04:05:44.064021 3440 tablet_service.cc:1467] Processing CreateTablet for tablet 55ab0cb20398435ebc28a993a65026b6 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 04:05:44.070149 3443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749844bfd19e4bdb8bd8aad14b132a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.070264 3433 tablet_service.cc:1467] Processing CreateTablet for tablet 13ab8d40e1ba45ad9549a392cbdd87b6 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 04:05:44.074530 3437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18319d34a9154a27946dda911923380a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.074353 3429 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6c24b5a0ce4e7c9941928a3a518f05 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 04:05:44.077225 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40f9b110c2e4169b055a1c0de83d19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.079789 3439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b09fbea2f484bf28173d95655b8a0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.083367 3433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ab8d40e1ba45ad9549a392cbdd87b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.085875 3429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6c24b5a0ce4e7c9941928a3a518f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.115267 3440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ab0cb20398435ebc28a993a65026b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.134956 3435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd58b8e6c32b4b1ba07225530f879797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.159425 3337 tablet_service.cc:1467] Processing CreateTablet for tablet 749844bfd19e4bdb8bd8aad14b132a78 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 04:05:44.169107 3333 tablet_service.cc:1467] Processing CreateTablet for tablet 6b09fbea2f484bf28173d95655b8a0a3 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 04:05:44.170178 3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749844bfd19e4bdb8bd8aad14b132a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.172022 3336 tablet_service.cc:1467] Processing CreateTablet for tablet 9323adac1819491f8d688196ca1a696a (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 04:05:44.170554 3331 tablet_service.cc:1467] Processing CreateTablet for tablet 18319d34a9154a27946dda911923380a (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 04:05:44.174746 3334 tablet_service.cc:1467] Processing CreateTablet for tablet 55ab0cb20398435ebc28a993a65026b6 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 04:05:44.189922 3330 tablet_service.cc:1467] Processing CreateTablet for tablet dd58b8e6c32b4b1ba07225530f879797 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 04:05:44.188381 3328 tablet_service.cc:1467] Processing CreateTablet for tablet 13ab8d40e1ba45ad9549a392cbdd87b6 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 04:05:44.193183 3333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b09fbea2f484bf28173d95655b8a0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.193794 3329 tablet_service.cc:1467] Processing CreateTablet for tablet e40f9b110c2e4169b055a1c0de83d19f (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 04:05:44.201303 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6c24b5a0ce4e7c9941928a3a518f05 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 04:05:44.205511 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9323adac1819491f8d688196ca1a696a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.209309 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ab0cb20398435ebc28a993a65026b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.212667 3327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6c24b5a0ce4e7c9941928a3a518f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.220876 3330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd58b8e6c32b4b1ba07225530f879797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.222908 3331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18319d34a9154a27946dda911923380a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.228987 3225 tablet_service.cc:1467] Processing CreateTablet for tablet 749844bfd19e4bdb8bd8aad14b132a78 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 04:05:44.234757 3328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ab8d40e1ba45ad9549a392cbdd87b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.240285 3223 tablet_service.cc:1467] Processing CreateTablet for tablet 9323adac1819491f8d688196ca1a696a (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 04:05:44.244350 3225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749844bfd19e4bdb8bd8aad14b132a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.252776 3219 tablet_service.cc:1467] Processing CreateTablet for tablet 18319d34a9154a27946dda911923380a (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 04:05:44.251127 3329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40f9b110c2e4169b055a1c0de83d19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.238075 3221 tablet_service.cc:1467] Processing CreateTablet for tablet 55ab0cb20398435ebc28a993a65026b6 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 04:05:44.258430 3223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9323adac1819491f8d688196ca1a696a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.257241 3220 tablet_service.cc:1467] Processing CreateTablet for tablet 6b09fbea2f484bf28173d95655b8a0a3 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 04:05:44.280582 3216 tablet_service.cc:1467] Processing CreateTablet for tablet e40f9b110c2e4169b055a1c0de83d19f (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 04:05:44.285581 3219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18319d34a9154a27946dda911923380a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.303081 3217 tablet_service.cc:1467] Processing CreateTablet for tablet dd58b8e6c32b4b1ba07225530f879797 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 04:05:44.303000 3215 tablet_service.cc:1467] Processing CreateTablet for tablet 13ab8d40e1ba45ad9549a392cbdd87b6 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 04:05:44.310117 3214 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6c24b5a0ce4e7c9941928a3a518f05 (DEFAULT_TABLE table=test-workload [id=2262dddbf8fe427486da51c2eef2b5d1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 04:05:44.314688 3517 tablet_bootstrap.cc:492] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:44.319034 3216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40f9b110c2e4169b055a1c0de83d19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.334625 3214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6c24b5a0ce4e7c9941928a3a518f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.346494 3215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ab8d40e1ba45ad9549a392cbdd87b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.363019 3217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd58b8e6c32b4b1ba07225530f879797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.389295 3517 tablet_bootstrap.cc:654] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.380582 3220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b09fbea2f484bf28173d95655b8a0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.419773 3221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ab0cb20398435ebc28a993a65026b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:05:44.440621 3274 heartbeater.cc:503] Master 127.2.248.126:34201 was elected leader, sending a full tablet report...
I20260806 04:05:44.456243 3518 tablet_bootstrap.cc:492] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:44.472913 3517 tablet_bootstrap.cc:492] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:44.477979 3517 ts_tablet_manager.cc:1397] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.021s
I20260806 04:05:44.483826 3517 raft_consensus.cc:348] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.489310 3517 raft_consensus.cc:374] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.489892 3517 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.490852 3517 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.516515 3517 ts_tablet_manager.cc:1428] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.038s user 0.018s sys 0.008s
I20260806 04:05:44.521425 3495 heartbeater.cc:503] Master 127.2.248.126:34201 was elected leader, sending a full tablet report...
I20260806 04:05:44.523121 3517 tablet_bootstrap.cc:492] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:44.523973 3518 tablet_bootstrap.cc:654] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
W20260806 04:05:44.527611 3496 tablet.cc:2369] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:05:44.527518 3520 tablet_bootstrap.cc:492] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:44.537235 3517 tablet_bootstrap.cc:654] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.553083 3517 tablet_bootstrap.cc:492] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:44.553524 3518 tablet_bootstrap.cc:492] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:44.553699 3517 ts_tablet_manager.cc:1397] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.006s
I20260806 04:05:44.554015 3518 ts_tablet_manager.cc:1397] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.118s user 0.012s sys 0.009s
I20260806 04:05:44.558825 3517 raft_consensus.cc:348] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.559875 3517 raft_consensus.cc:374] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.559454 3518 raft_consensus.cc:348] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.560258 3517 raft_consensus.cc:729] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.561216 3517 consensus_queue.cc:260] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.562963 3520 tablet_bootstrap.cc:654] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.560344 3518 raft_consensus.cc:374] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.563839 3518 raft_consensus.cc:729] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.563649 3517 ts_tablet_manager.cc:1428] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260806 04:05:44.565649 3517 tablet_bootstrap.cc:492] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:44.566828 3518 consensus_queue.cc:260] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.575346 3517 tablet_bootstrap.cc:654] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.578644 3384 heartbeater.cc:503] Master 127.2.248.126:34201 was elected leader, sending a full tablet report...
I20260806 04:05:44.579736 3518 ts_tablet_manager.cc:1428] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.025s user 0.010s sys 0.005s
I20260806 04:05:44.580765 3518 tablet_bootstrap.cc:492] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:44.593642 3520 tablet_bootstrap.cc:492] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:44.594281 3520 ts_tablet_manager.cc:1397] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.011s
I20260806 04:05:44.597416 3517 tablet_bootstrap.cc:492] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:44.598063 3517 ts_tablet_manager.cc:1397] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.011s
I20260806 04:05:44.598448 3520 raft_consensus.cc:348] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.599620 3520 raft_consensus.cc:374] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.600021 3520 raft_consensus.cc:729] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.600893 3520 consensus_queue.cc:260] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.603469 3518 tablet_bootstrap.cc:654] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.607800 3517 raft_consensus.cc:348] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.609688 3517 raft_consensus.cc:374] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.610230 3517 raft_consensus.cc:729] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.615422 3517 consensus_queue.cc:260] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.621518 3520 ts_tablet_manager.cc:1428] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.027s user 0.004s sys 0.015s
I20260806 04:05:44.623036 3520 tablet_bootstrap.cc:492] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:44.631498 3517 ts_tablet_manager.cc:1428] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.033s user 0.009s sys 0.006s
I20260806 04:05:44.633143 3517 tablet_bootstrap.cc:492] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:44.634496 3520 tablet_bootstrap.cc:654] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.646013 3518 tablet_bootstrap.cc:492] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:44.648924 3518 ts_tablet_manager.cc:1397] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.068s user 0.034s sys 0.000s
I20260806 04:05:44.654767 3520 tablet_bootstrap.cc:492] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:44.655507 3520 ts_tablet_manager.cc:1397] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.010s
I20260806 04:05:44.656927 3518 raft_consensus.cc:348] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.657766 3517 tablet_bootstrap.cc:654] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.658186 3518 raft_consensus.cc:374] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.658676 3518 raft_consensus.cc:729] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.660005 3518 consensus_queue.cc:260] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.661057 3520 raft_consensus.cc:348] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.663089 3520 raft_consensus.cc:374] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.663769 3520 raft_consensus.cc:729] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.665525 3518 ts_tablet_manager.cc:1428] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.016s user 0.003s sys 0.008s
I20260806 04:05:44.664841 3520 consensus_queue.cc:260] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.666729 3518 tablet_bootstrap.cc:492] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:44.667691 3520 ts_tablet_manager.cc:1428] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260806 04:05:44.669164 3520 tablet_bootstrap.cc:492] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:44.676353 3517 tablet_bootstrap.cc:492] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:44.676934 3517 ts_tablet_manager.cc:1397] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.012s
I20260806 04:05:44.677606 3520 tablet_bootstrap.cc:654] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.680260 3517 raft_consensus.cc:348] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.681164 3517 raft_consensus.cc:374] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.681511 3517 raft_consensus.cc:729] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.683013 3518 tablet_bootstrap.cc:654] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.682471 3517 consensus_queue.cc:260] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.692317 3517 ts_tablet_manager.cc:1428] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260806 04:05:44.693778 3517 tablet_bootstrap.cc:492] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:44.697558 3520 tablet_bootstrap.cc:492] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:44.698114 3520 ts_tablet_manager.cc:1397] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.011s
I20260806 04:05:44.701629 3530 raft_consensus.cc:482] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:44.701196 3520 raft_consensus.cc:348] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.703015 3520 raft_consensus.cc:374] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.702555 3530 raft_consensus.cc:504] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.703411 3520 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.704285 3520 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.706418 3530 leader_election.cc:288] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:44.711090 3534 raft_consensus.cc:482] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:44.712323 3534 raft_consensus.cc:504] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.711444 3520 ts_tablet_manager.cc:1428] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.013s user 0.005s sys 0.005s
I20260806 04:05:44.718837 3517 tablet_bootstrap.cc:654] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.728263 3520 tablet_bootstrap.cc:492] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:44.728232 3534 leader_election.cc:288] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:44.743204 3518 tablet_bootstrap.cc:492] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:44.743963 3518 ts_tablet_manager.cc:1397] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.011s
I20260806 04:05:44.757133 3518 raft_consensus.cc:348] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.761590 3518 raft_consensus.cc:374] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.762033 3518 raft_consensus.cc:729] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.764410 3518 consensus_queue.cc:260] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.769423 3520 tablet_bootstrap.cc:654] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.784008 3518 ts_tablet_manager.cc:1428] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.040s user 0.009s sys 0.002s
I20260806 04:05:44.785329 3518 tablet_bootstrap.cc:492] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:44.812913 3517 tablet_bootstrap.cc:492] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:44.813599 3517 ts_tablet_manager.cc:1397] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.024s
I20260806 04:05:44.820096 3518 tablet_bootstrap.cc:654] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.824257 3517 raft_consensus.cc:348] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.825700 3517 raft_consensus.cc:374] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.826198 3517 raft_consensus.cc:729] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.832131 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:44.833461 3355 raft_consensus.cc:2454] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 0.
I20260806 04:05:44.835405 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:44.837860 3190 leader_election.cc:302] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:44.839581 3534 raft_consensus.cc:2781] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:44.840489 3534 raft_consensus.cc:482] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:44.841183 3534 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:44.842751 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:44.843640 3518 tablet_bootstrap.cc:492] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:44.844115 3518 ts_tablet_manager.cc:1397] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.010s
I20260806 04:05:44.846837 3522 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:05:44.847404 3300 leader_election.cc:341] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.827461 3517 consensus_queue.cc:260] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.848201 3522 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.849203 3460 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:44.857520 3522 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:44.867877 3517 ts_tablet_manager.cc:1428] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.054s user 0.011s sys 0.006s
I20260806 04:05:44.869236 3517 tablet_bootstrap.cc:492] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:44.865398 3534 raft_consensus.cc:504] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.872829 3520 tablet_bootstrap.cc:492] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:44.873385 3520 ts_tablet_manager.cc:1397] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.145s user 0.017s sys 0.016s
I20260806 04:05:44.876417 3518 raft_consensus.cc:348] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.876991 3520 raft_consensus.cc:348] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.877449 3518 raft_consensus.cc:374] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.877797 3520 raft_consensus.cc:374] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:44.877931 3518 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.878157 3520 raft_consensus.cc:729] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:44.879109 3518 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.880313 3534 leader_election.cc:288] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:44.882129 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:44.882951 3355 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:44.886200 3520 consensus_queue.cc:260] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.887933 3518 ts_tablet_manager.cc:1428] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.043s user 0.009s sys 0.001s
I20260806 04:05:44.893399 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:44.893800 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:44.895748 3520 ts_tablet_manager.cc:1428] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20260806 04:05:44.897084 3520 tablet_bootstrap.cc:492] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
W20260806 04:05:44.896991 3190 leader_election.cc:341] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:05:44.897835 3190 leader_election.cc:341] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.906570 3518 tablet_bootstrap.cc:492] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:44.908118 3534 raft_consensus.cc:482] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:44.908815 3534 raft_consensus.cc:504] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.910514 3355 raft_consensus.cc:2454] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 1.
I20260806 04:05:44.912966 3299 leader_election.cc:302] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters: 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:05:44.913322 3190 leader_election.cc:302] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters: 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:44.912464 3522 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:44.914819 3530 raft_consensus.cc:2781] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:44.914438 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:44.915510 3530 raft_consensus.cc:482] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:44.915316 3522 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:44.915915 3530 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:44.919395 3522 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:05:44.922573 3534 leader_election.cc:288] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:44.928988 3534 raft_consensus.cc:2781] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:44.930954 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
W20260806 04:05:44.933190 3190 leader_election.cc:341] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.933480 3517 tablet_bootstrap.cc:654] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
W20260806 04:05:44.933965 3190 leader_election.cc:341] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.932909 3534 raft_consensus.cc:686] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:44.936600 3534 consensus_queue.cc:237] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.937657 3530 raft_consensus.cc:504] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.940549 3518 tablet_bootstrap.cc:654] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.934435 3190 leader_election.cc:302] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a; no voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:44.942633 3522 raft_consensus.cc:482] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:44.943369 3522 raft_consensus.cc:504] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.944546 3555 raft_consensus.cc:2726] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:05:44.947114 3522 leader_election.cc:288] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:44.951198 3520 tablet_bootstrap.cc:654] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:44.955370 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
W20260806 04:05:44.957327 3300 leader_election.cc:341] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.957420 3555 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:44.959131 3530 leader_election.cc:288] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:44.958280 3555 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:44.962568 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:44.964978 3237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:44.967026 3555 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:44.966291 3460 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:44.969796 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:44.971192 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:44.971907 3459 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 0.
I20260806 04:05:44.975170 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:44.976827 3238 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 0.
I20260806 04:05:44.964938 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
W20260806 04:05:44.980522 3190 leader_election.cc:341] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.980667 3239 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 0.
I20260806 04:05:44.983057 3190 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters: 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:44.983976 3555 raft_consensus.cc:2781] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:44.984404 3555 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:05:44.985451 3404 leader_election.cc:341] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:44.986945 3404 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:44.985517 3555 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:44.989147 3522 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:44.989941 3522 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:44.990391 3522 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:44.992656 3404 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:45.027762 3518 tablet_bootstrap.cc:492] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:45.029198 3518 ts_tablet_manager.cc:1397] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.123s user 0.021s sys 0.013s
I20260806 04:05:45.025617 3086 catalog_manager.cc:5495] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.028028 3561 raft_consensus.cc:2781] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:45.034453 3460 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.035846 3299 leader_election.cc:302] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters: 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:05:45.037127 3530 raft_consensus.cc:2781] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.032495 3555 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.032495 3561 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.046705 3522 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:45.047752 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.042400 3561 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.052285 3517 tablet_bootstrap.cc:492] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
W20260806 04:05:45.052520 3403 leader_election.cc:341] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.052959 3517 ts_tablet_manager.cc:1397] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.184s user 0.031s sys 0.030s
W20260806 04:05:45.058951 3162 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:05:45.061022 3354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.063704 3354 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 0.
I20260806 04:05:45.055423 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.061383 3518 raft_consensus.cc:348] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.076429 3354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
W20260806 04:05:45.076875 3403 leader_election.cc:341] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.073531 3518 raft_consensus.cc:374] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.077340 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:45.079856 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:45.073158 3530 raft_consensus.cc:686] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:45.075220 3555 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
W20260806 04:05:45.080231 3190 leader_election.cc:341] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.078845 3518 raft_consensus.cc:729] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.078536 3522 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:05:45.080458 3352 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.079125 3403 leader_election.cc:302] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3; no voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:45.083937 3518 consensus_queue.cc:260] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.086308 3520 tablet_bootstrap.cc:492] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:45.085686 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.077773 3517 raft_consensus.cc:348] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.082630 3530 consensus_queue.cc:237] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.088122 3239 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.089136 3517 raft_consensus.cc:374] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.088318 3570 raft_consensus.cc:482] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.088423 3522 raft_consensus.cc:2726] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.089214 3520 ts_tablet_manager.cc:1397] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.192s user 0.024s sys 0.036s
I20260806 04:05:45.093137 3517 raft_consensus.cc:729] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.097985 3563 raft_consensus.cc:482] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.094332 3570 raft_consensus.cc:504] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.101548 3563 raft_consensus.cc:504] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.108152 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
W20260806 04:05:45.111299 3300 leader_election.cc:341] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.103299 3517 consensus_queue.cc:260] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.113042 3352 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 1.
I20260806 04:05:45.114104 3563 leader_election.cc:288] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:45.114409 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.116612 3354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.119517 3517 ts_tablet_manager.cc:1428] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.066s user 0.007s sys 0.004s
I20260806 04:05:45.120018 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.123693 3567 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:05:45.124471 3404 leader_election.cc:341] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:05:45.125443 3299 leader_election.cc:341] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.126775 3518 ts_tablet_manager.cc:1428] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.097s user 0.008s sys 0.019s
I20260806 04:05:45.126502 3299 leader_election.cc:302] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9040ee4726534f72b52b4c4fd0188fc3; no voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:45.128015 3518 tablet_bootstrap.cc:492] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:45.128086 3520 raft_consensus.cc:348] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.129184 3520 raft_consensus.cc:374] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:05:45.125330 3403 leader_election.cc:341] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.132614 3571 raft_consensus.cc:2726] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.124416 3567 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:45.129696 3520 raft_consensus.cc:729] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.132865 3239 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 1.
I20260806 04:05:45.120657 3570 leader_election.cc:288] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.135715 3404 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:45.138779 3563 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.132608 3403 leader_election.cc:302] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3; no voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:45.135228 3520 consensus_queue.cc:260] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.136932 3517 tablet_bootstrap.cc:492] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:45.139967 3563 raft_consensus.cc:686] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 1 LEADER]: Becoming Leader. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:05:45.140720 3522 raft_consensus.cc:2726] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.159601 3561 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.161538 3460 raft_consensus.cc:2379] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d3a226c500b4528a2dea7cd30910d6a in current term 1: Already voted for candidate 8983098d2f3843199e31fe0c278e12a3 in this term.
I20260806 04:05:45.160059 3563 consensus_queue.cc:237] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:45.166553 3190 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a; no voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:45.167519 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.168263 3555 raft_consensus.cc:2726] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.173143 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.174233 3239 raft_consensus.cc:2379] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8983098d2f3843199e31fe0c278e12a3 in current term 1: Already voted for candidate 2d3a226c500b4528a2dea7cd30910d6a in this term.
I20260806 04:05:45.174921 3087 catalog_manager.cc:5495] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.188434 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.189414 3239 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.191148 3300 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters: 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:45.191784 3567 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789), 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:05:45.193164 3570 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:05:45.193635 3570 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.194072 3570 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:45.198179 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
W20260806 04:05:45.200168 3403 leader_election.cc:341] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:05:45.196687 3520 ts_tablet_manager.cc:1428] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.102s user 0.007s sys 0.006s
I20260806 04:05:45.201287 3403 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3; no voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:45.202895 3563 raft_consensus.cc:2726] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.196074 3561 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:45.207062 3520 tablet_bootstrap.cc:492] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:45.208923 3085 catalog_manager.cc:5495] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "8983098d2f3843199e31fe0c278e12a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.215867 3570 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:45.221685 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.222095 3520 tablet_bootstrap.cc:654] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.220597 3570 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 election: Requested vote from peers 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789), 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:05:45.222687 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:45.224035 3518 tablet_bootstrap.cc:654] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.222378 3239 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:45.224057 3517 tablet_bootstrap.cc:654] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.246412 3239 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 2.
I20260806 04:05:45.253834 3300 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters: 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:45.255048 3561 raft_consensus.cc:482] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.255676 3570 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:45.255841 3561 raft_consensus.cc:504] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.256531 3570 raft_consensus.cc:686] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:45.257790 3570 consensus_queue.cc:237] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:45.264153 3561 leader_election.cc:288] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:45.265784 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6c24b5a0ce4e7c9941928a3a518f05" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.268818 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6c24b5a0ce4e7c9941928a3a518f05" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.270640 3352 raft_consensus.cc:2454] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 0.
I20260806 04:05:45.272462 3403 leader_election.cc:302] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.273542 3561 raft_consensus.cc:2781] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:45.273970 3561 raft_consensus.cc:482] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.274369 3561 raft_consensus.cc:3037] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.274880 3239 raft_consensus.cc:2454] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 0.
I20260806 04:05:45.285569 3518 tablet_bootstrap.cc:492] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:45.285744 3520 tablet_bootstrap.cc:492] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:45.286190 3518 ts_tablet_manager.cc:1397] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.158s user 0.015s sys 0.025s
I20260806 04:05:45.286423 3520 ts_tablet_manager.cc:1397] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.024s
I20260806 04:05:45.288110 3517 tablet_bootstrap.cc:492] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:45.288844 3517 ts_tablet_manager.cc:1397] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.152s user 0.014s sys 0.024s
I20260806 04:05:45.288826 3561 raft_consensus.cc:504] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.292644 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6c24b5a0ce4e7c9941928a3a518f05" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.293403 3239 raft_consensus.cc:3037] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.293974 3561 leader_election.cc:288] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:45.294287 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6c24b5a0ce4e7c9941928a3a518f05" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:45.294991 3352 raft_consensus.cc:3037] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.292059 3086 catalog_manager.cc:5495] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 1 to 2, leader changed from 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.296219 3517 raft_consensus.cc:348] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.298400 3517 raft_consensus.cc:374] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.299556 3518 raft_consensus.cc:348] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.300709 3518 raft_consensus.cc:374] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.300328 3517 raft_consensus.cc:729] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.301543 3517 consensus_queue.cc:260] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.302752 3352 raft_consensus.cc:2454] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 1.
I20260806 04:05:45.296634 3520 raft_consensus.cc:348] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.306836 3520 raft_consensus.cc:374] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.307227 3520 raft_consensus.cc:729] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.308043 3520 consensus_queue.cc:260] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.310393 3520 ts_tablet_manager.cc:1428] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.024s user 0.001s sys 0.003s
I20260806 04:05:45.311571 3520 tablet_bootstrap.cc:492] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:45.314157 3239 raft_consensus.cc:2454] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 1.
I20260806 04:05:45.305326 3517 ts_tablet_manager.cc:1428] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260806 04:05:45.315568 3517 tablet_bootstrap.cc:492] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:45.315651 3404 leader_election.cc:302] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:45.302456 3518 raft_consensus.cc:729] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.323603 3561 raft_consensus.cc:2781] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.323328 3518 consensus_queue.cc:260] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.324124 3561 raft_consensus.cc:686] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 1 LEADER]: Becoming Leader. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:05:45.324987 3561 consensus_queue.cc:237] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.326349 3518 ts_tablet_manager.cc:1428] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.040s user 0.008s sys 0.000s
I20260806 04:05:45.327486 3518 tablet_bootstrap.cc:492] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:45.342343 3520 tablet_bootstrap.cc:654] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.343870 3517 tablet_bootstrap.cc:654] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.344107 3460 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.348233 3518 tablet_bootstrap.cc:654] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.364413 3555 raft_consensus.cc:482] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.365216 3555 raft_consensus.cc:504] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.369518 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.371497 3555 leader_election.cc:288] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.373104 3534 consensus_queue.cc:1035] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.374392 3517 tablet_bootstrap.cc:492] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:45.375057 3517 ts_tablet_manager.cc:1397] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.020s
I20260806 04:05:45.376698 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.379288 3460 raft_consensus.cc:2379] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d3a226c500b4528a2dea7cd30910d6a in current term 1: Already voted for candidate 9040ee4726534f72b52b4c4fd0188fc3 in this term.
I20260806 04:05:45.379177 3517 raft_consensus.cc:348] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.375836 3087 catalog_manager.cc:5495] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "8983098d2f3843199e31fe0c278e12a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: HEALTHY } } }
I20260806 04:05:45.381798 3517 raft_consensus.cc:374] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.383373 3517 raft_consensus.cc:729] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.383105 3190 leader_election.cc:302] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a; no voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:45.384238 3517 consensus_queue.cc:260] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.386158 3534 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.388249 3517 ts_tablet_manager.cc:1428] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260806 04:05:45.389204 3517 tablet_bootstrap.cc:492] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: Bootstrap starting.
I20260806 04:05:45.396943 3534 raft_consensus.cc:2726] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.401467 3517 tablet_bootstrap.cc:654] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.405884 3520 tablet_bootstrap.cc:492] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:45.407147 3518 tablet_bootstrap.cc:492] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:45.407778 3520 ts_tablet_manager.cc:1397] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.096s user 0.024s sys 0.028s
I20260806 04:05:45.407778 3518 ts_tablet_manager.cc:1397] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.000s
I20260806 04:05:45.417570 3520 raft_consensus.cc:348] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.424007 3520 raft_consensus.cc:374] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.424914 3520 raft_consensus.cc:729] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.427488 3520 consensus_queue.cc:260] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.444270 3534 consensus_queue.cc:1035] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.445163 3517 tablet_bootstrap.cc:492] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: No bootstrap required, opened a new log
I20260806 04:05:45.445639 3517 ts_tablet_manager.cc:1397] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.023s
I20260806 04:05:45.442912 3518 raft_consensus.cc:348] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.451751 3518 raft_consensus.cc:374] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.452126 3518 raft_consensus.cc:729] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.452903 3518 consensus_queue.cc:260] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.457055 3520 ts_tablet_manager.cc:1428] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.049s user 0.010s sys 0.000s
I20260806 04:05:45.460482 3517 raft_consensus.cc:348] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.460995 3517 raft_consensus.cc:374] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.461258 3517 raft_consensus.cc:729] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.461822 3517 consensus_queue.cc:260] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.466922 3517 ts_tablet_manager.cc:1428] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: Time spent starting tablet: real 0.021s user 0.005s sys 0.003s
I20260806 04:05:45.470726 3520 tablet_bootstrap.cc:492] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:45.477723 3518 ts_tablet_manager.cc:1428] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.055s user 0.004s sys 0.008s
I20260806 04:05:45.483629 3518 tablet_bootstrap.cc:492] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:45.478785 3520 tablet_bootstrap.cc:654] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.491405 3518 tablet_bootstrap.cc:654] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.508428 3518 tablet_bootstrap.cc:492] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:45.509019 3518 ts_tablet_manager.cc:1397] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.013s
I20260806 04:05:45.510774 3567 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:45.514782 3520 tablet_bootstrap.cc:492] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:45.515377 3520 ts_tablet_manager.cc:1397] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.010s
I20260806 04:05:45.517879 3518 raft_consensus.cc:348] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.518967 3518 raft_consensus.cc:374] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.519389 3518 raft_consensus.cc:729] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.520381 3518 consensus_queue.cc:260] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.523241 3518 ts_tablet_manager.cc:1428] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260806 04:05:45.524470 3518 tablet_bootstrap.cc:492] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: Bootstrap starting.
I20260806 04:05:45.527983 3520 raft_consensus.cc:348] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.528851 3520 raft_consensus.cc:374] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.529132 3520 raft_consensus.cc:729] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.529860 3520 consensus_queue.cc:260] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.534567 3518 tablet_bootstrap.cc:654] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.543361 3520 ts_tablet_manager.cc:1428] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.028s user 0.004s sys 0.004s
I20260806 04:05:45.546537 3520 tablet_bootstrap.cc:492] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: Bootstrap starting.
I20260806 04:05:45.559674 3520 tablet_bootstrap.cc:654] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: Neither blocks nor log segments found. Creating new log.
I20260806 04:05:45.559768 3518 tablet_bootstrap.cc:492] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: No bootstrap required, opened a new log
I20260806 04:05:45.560465 3518 ts_tablet_manager.cc:1397] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.009s
I20260806 04:05:45.566609 3530 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.578007 3518 raft_consensus.cc:348] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.579772 3518 raft_consensus.cc:374] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.583053 3518 raft_consensus.cc:729] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.584549 3518 consensus_queue.cc:260] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.585945 3518 ts_tablet_manager.cc:1428] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260806 04:05:45.588717 3520 tablet_bootstrap.cc:492] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: No bootstrap required, opened a new log
I20260806 04:05:45.589252 3520 ts_tablet_manager.cc:1397] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent bootstrapping tablet: real 0.043s user 0.019s sys 0.005s
I20260806 04:05:45.590298 3352 raft_consensus.cc:1226] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Rejecting Update request from peer 8983098d2f3843199e31fe0c278e12a3 for earlier term 1. Current term is 2. Ops: []
I20260806 04:05:45.591992 3520 raft_consensus.cc:348] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.592180 3561 consensus_queue.cc:1046] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.592866 3520 raft_consensus.cc:374] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:05:45.593256 3561 consensus_queue.cc:1225] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Peer 9040ee4726534f72b52b4c4fd0188fc3 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260806 04:05:45.594125 3563 raft_consensus.cc:3032] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:05:45.594470 3563 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:05:45.593242 3520 raft_consensus.cc:729] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Initialized, Role: FOLLOWER
I20260806 04:05:45.595400 3520 consensus_queue.cc:260] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.595362 3563 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:45.597097 3563 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:45.598420 3520 ts_tablet_manager.cc:1428] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260806 04:05:45.614622 3571 raft_consensus.cc:482] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.615446 3571 raft_consensus.cc:504] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.616160 3596 raft_consensus.cc:482] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.616843 3596 raft_consensus.cc:504] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.619272 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.620378 3459 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:45.620746 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.621302 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.621531 3239 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:45.622025 3460 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 0.
I20260806 04:05:45.621845 3299 leader_election.cc:302] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.619154 3596 leader_election.cc:288] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.623415 3567 raft_consensus.cc:2781] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:45.623469 3190 leader_election.cc:302] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:45.623973 3567 raft_consensus.cc:482] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.624432 3567 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.624565 3596 raft_consensus.cc:2781] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:45.625000 3596 raft_consensus.cc:482] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.625099 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:45.625396 3596 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.625814 3352 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 0.
I20260806 04:05:45.627247 3571 leader_election.cc:288] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.628350 3571 raft_consensus.cc:482] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.629066 3571 raft_consensus.cc:504] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.632850 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.633255 3567 raft_consensus.cc:504] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.633761 3239 raft_consensus.cc:2454] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:45.634333 3596 raft_consensus.cc:504] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.635304 3300 leader_election.cc:302] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.636087 3567 leader_election.cc:288] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.636631 3570 raft_consensus.cc:2781] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:45.637022 3570 raft_consensus.cc:482] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.637581 3570 raft_consensus.cc:3037] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.637478 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:45.638180 3352 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.639374 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:45.640102 3460 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.642391 3596 leader_election.cc:288] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.643586 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.644317 3238 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.644556 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:45.647603 3459 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.647764 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.648518 3454 raft_consensus.cc:2454] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:45.648690 3571 leader_election.cc:288] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.653415 3570 raft_consensus.cc:504] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.655508 3352 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 1.
I20260806 04:05:45.655725 3238 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.657217 3190 leader_election.cc:302] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.658187 3300 leader_election.cc:302] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.658969 3571 raft_consensus.cc:2781] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.659580 3596 raft_consensus.cc:2781] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.659683 3571 raft_consensus.cc:686] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:45.661113 3596 raft_consensus.cc:686] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:45.660959 3571 consensus_queue.cc:237] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.661468 3459 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.662192 3596 consensus_queue.cc:237] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.663756 3460 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 1.
I20260806 04:05:45.664748 3570 leader_election.cc:288] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.666296 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:45.666942 3454 raft_consensus.cc:3037] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.658314 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.668814 3238 raft_consensus.cc:3037] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.674947 3454 raft_consensus.cc:2454] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.676139 3299 leader_election.cc:302] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.677349 3571 raft_consensus.cc:2781] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.678072 3571 raft_consensus.cc:686] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:45.679033 3571 consensus_queue.cc:237] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.678589 3085 catalog_manager.cc:5495] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.690307 3238 raft_consensus.cc:2454] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.694998 3238 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 04:05:45.696228 3087 catalog_manager.cc:5495] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.696771 3571 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.696045 3085 catalog_manager.cc:5495] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:45.717758 3454 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260806 04:05:45.718338 3454 pending_rounds.cc:76] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Aborting all ops after (but not including) 0
I20260806 04:05:45.718815 3454 pending_rounds.cc:97] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260806 04:05:45.719153 3454 raft_consensus.cc:2866] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:05:45.721906 3567 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.837167 3567 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.837668 3567 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.839540 3567 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.840794 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.840794 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.841835 3454 raft_consensus.cc:2379] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9040ee4726534f72b52b4c4fd0188fc3 in current term 1: Already voted for candidate 8983098d2f3843199e31fe0c278e12a3 in this term.
I20260806 04:05:45.841856 3238 raft_consensus.cc:2379] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9040ee4726534f72b52b4c4fd0188fc3 in current term 1: Already voted for candidate 2d3a226c500b4528a2dea7cd30910d6a in this term.
I20260806 04:05:45.843258 3300 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9040ee4726534f72b52b4c4fd0188fc3; no voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:45.844108 3567 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.849454 3567 raft_consensus.cc:2726] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:05:45.863713 3563 consensus_queue.cc:1035] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:45.881055 3622 consensus_queue.cc:1035] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:45.945813 3567 raft_consensus.cc:482] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:45.946344 3567 raft_consensus.cc:504] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.948458 3567 leader_election.cc:288] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.949213 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a" is_pre_election: true
I20260806 04:05:45.949553 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:45.949813 3238 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:45.950202 3454 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 0.
I20260806 04:05:45.950947 3300 leader_election.cc:302] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.951713 3567 raft_consensus.cc:2781] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:05:45.952020 3567 raft_consensus.cc:482] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:45.952292 3567 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.957870 3567 raft_consensus.cc:504] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.959677 3567 leader_election.cc:288] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:45.960565 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:45.961014 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:45.961179 3238 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.961656 3454 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:05:45.968448 3454 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.968448 3238 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 1.
I20260806 04:05:45.969952 3300 leader_election.cc:302] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:45.970860 3567 raft_consensus.cc:2781] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:05:45.971514 3567 raft_consensus.cc:686] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:45.972537 3567 consensus_queue.cc:237] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:45.982239 3085 catalog_manager.cc:5495] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
W20260806 04:05:46.069505 3162 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:05:46.106099 3567 consensus_queue.cc:1035] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:46.106962 3571 consensus_queue.cc:1035] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:46.114981 3567 consensus_queue.cc:1035] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:46.148896 3567 consensus_queue.cc:1035] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:46.151522 3596 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:46.194912 3555 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:46.478991 3530 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:46.479751 3571 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:05:46.833493 3646 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:05:46.833959 3646 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:46.835732 3646 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:46.836705 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3" is_pre_election: true
I20260806 04:05:46.836820 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3" is_pre_election: true
I20260806 04:05:46.837513 3352 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 1.
I20260806 04:05:46.837627 3460 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 1.
I20260806 04:05:46.838670 3190 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:46.839366 3646 raft_consensus.cc:2781] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:05:46.839737 3646 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:05:46.840018 3646 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:46.845052 3646 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:46.846624 3646 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:46.847545 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:46.847710 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:46.848209 3352 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:46.848276 3460 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:46.854928 3460 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:46.854928 3352 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:46.856313 3190 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:46.857040 3646 raft_consensus.cc:2781] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:46.857523 3646 raft_consensus.cc:686] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:46.858157 3646 consensus_queue.cc:237] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:46.865463 3087 catalog_manager.cc:5495] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 0 to 2, leader changed from <none> to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.299973 3460 raft_consensus.cc:1261] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 04:05:47.301205 3646 consensus_queue.cc:1035] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:47.311576 3352 raft_consensus.cc:1261] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 04:05:47.314381 3646 consensus_queue.cc:1035] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:05:47.406220 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:57868
I20260806 04:05:47.406937 3352 raft_consensus.cc:595] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:05:47.411558 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9323adac1819491f8d688196ca1a696a"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:57868
I20260806 04:05:47.412052 3352 raft_consensus.cc:595] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:05:47.416996 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:57868
I20260806 04:05:47.417665 3352 raft_consensus.cc:595] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:05:47.422003 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:57868
I20260806 04:05:47.422514 3352 raft_consensus.cc:595] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:05:47.438295 3666 raft_consensus.cc:979] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.438571 3238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18319d34a9154a27946dda911923380a"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:44414
I20260806 04:05:47.438967 3530 raft_consensus.cc:1067] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.439286 3238 raft_consensus.cc:595] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:47.441238 3238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:05:47.441852 3238 raft_consensus.cc:482] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.442128 3238 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.446805 3238 raft_consensus.cc:504] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.448709 3238 leader_election.cc:288] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:47.449512 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:47.449868 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.450153 3352 raft_consensus.cc:3032] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:05:47.450444 3352 raft_consensus.cc:729] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.450495 3460 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.450206 3239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:44414
I20260806 04:05:47.451067 3239 raft_consensus.cc:595] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:47.451196 3352 consensus_queue.cc:260] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.452418 3352 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.457145 3460 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:47.458081 3352 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:47.458191 3239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:44414
I20260806 04:05:47.458858 3239 raft_consensus.cc:595] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:05:47.459421 3190 leader_election.cc:302] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:47.460336 3654 raft_consensus.cc:2781] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:47.460801 3654 raft_consensus.cc:686] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.461599 3654 consensus_queue.cc:237] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.468524 3085 catalog_manager.cc:5495] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 1 to 2, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.572162 3646 raft_consensus.cc:979] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:05:47.572692 3644 raft_consensus.cc:1067] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:05:47.574286 3354 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57860
I20260806 04:05:47.574836 3354 raft_consensus.cc:482] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.575187 3354 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.575582 3654 raft_consensus.cc:979] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:05:47.575992 3644 raft_consensus.cc:1067] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:05:47.577322 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18319d34a9154a27946dda911923380a"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57860
I20260806 04:05:47.577782 3352 raft_consensus.cc:482] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.578063 3352 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.580102 3354 raft_consensus.cc:504] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.582120 3354 leader_election.cc:288] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:47.582563 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:47.582785 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.583235 3239 raft_consensus.cc:3032] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:05:47.583042 3352 raft_consensus.cc:504] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.583370 3460 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.583632 3239 raft_consensus.cc:729] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.584478 3239 consensus_queue.cc:260] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.585408 3352 leader_election.cc:288] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:47.585865 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:47.586493 3238 raft_consensus.cc:3032] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:05:47.586812 3238 raft_consensus.cc:729] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.587477 3238 consensus_queue.cc:260] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.588385 3238 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.589192 3460 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 2.
I20260806 04:05:47.590157 3299 leader_election.cc:302] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:47.590683 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.590962 3239 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.591073 3530 raft_consensus.cc:2781] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:47.591374 3460 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.591537 3530 raft_consensus.cc:686] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.592464 3530 consensus_queue.cc:237] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.597451 3239 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 2.
I20260806 04:05:47.597601 3460 raft_consensus.cc:2454] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 2.
I20260806 04:05:47.598817 3299 leader_election.cc:302] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:47.599722 3530 raft_consensus.cc:2781] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:47.600244 3530 raft_consensus.cc:686] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.600989 3530 consensus_queue.cc:237] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.601733 3085 catalog_manager.cc:5495] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 1 to 2, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.603513 3238 raft_consensus.cc:2454] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 2.
I20260806 04:05:47.614575 3666 raft_consensus.cc:979] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.615100 3530 raft_consensus.cc:1067] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.614533 3085 catalog_manager.cc:5495] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 1 to 2, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.616525 3238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9323adac1819491f8d688196ca1a696a"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:05:47.616999 3238 raft_consensus.cc:482] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.617282 3238 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.622604 3238 raft_consensus.cc:504] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.624429 3238 leader_election.cc:288] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:47.625326 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:47.625530 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.625855 3352 raft_consensus.cc:3032] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:05:47.626099 3460 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.626161 3352 raft_consensus.cc:729] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.627007 3352 consensus_queue.cc:260] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.628007 3352 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.631497 3460 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:47.632790 3190 leader_election.cc:302] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:47.633286 3352 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:47.633723 3644 raft_consensus.cc:2781] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:47.634191 3644 raft_consensus.cc:686] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.635020 3644 consensus_queue.cc:237] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.635848 3666 raft_consensus.cc:979] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.636322 3530 raft_consensus.cc:1067] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.637784 3238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:05:47.638305 3238 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.638747 3238 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:47.643234 3085 catalog_manager.cc:5495] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 1 to 2, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.645823 3238 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:47.647401 3238 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 3 election: Requested vote from peers 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:47.648293 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.648628 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:47.648854 3460 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:47.649197 3352 raft_consensus.cc:3032] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:05:47.649483 3352 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.650149 3352 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:47.651116 3352 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:47.653534 3460 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 3.
I20260806 04:05:47.654534 3190 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:47.655296 3644 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:05:47.655730 3644 raft_consensus.cc:686] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.656373 3352 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 3.
I20260806 04:05:47.656437 3644 consensus_queue.cc:237] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:05:47.663226 3666 raft_consensus.cc:979] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.663717 3530 raft_consensus.cc:1067] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:05:47.663975 3085 catalog_manager.cc:5495] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 2 to 3, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } }
I20260806 04:05:47.665196 3238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:05:47.665685 3238 raft_consensus.cc:482] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.665998 3238 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.672371 3238 raft_consensus.cc:504] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.674116 3238 leader_election.cc:288] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:47.675046 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:47.675372 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.675633 3352 raft_consensus.cc:3032] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:05:47.675925 3460 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.675981 3352 raft_consensus.cc:729] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.676774 3352 consensus_queue.cc:260] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.677706 3352 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:05:47.681026 3460 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:47.682065 3190 leader_election.cc:302] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:47.682631 3352 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 2.
I20260806 04:05:47.682835 3644 raft_consensus.cc:2781] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:05:47.683401 3644 raft_consensus.cc:686] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.684301 3644 consensus_queue.cc:237] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.691772 3085 catalog_manager.cc:5495] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 1 to 2, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.731343 3654 raft_consensus.cc:979] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:05:47.731844 3644 raft_consensus.cc:1067] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:05:47.733441 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57860
I20260806 04:05:47.734045 3352 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:47.734409 3352 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:47.739266 3352 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.740797 3352 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 3 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:05:47.741799 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:47.741963 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:05:47.742396 3238 raft_consensus.cc:3032] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:05:47.742624 3460 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:47.742857 3238 raft_consensus.cc:729] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:47.743562 3238 consensus_queue.cc:260] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.744547 3238 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:47.747910 3460 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 3.
I20260806 04:05:47.749014 3299 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:05:47.749115 3238 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 3.
I20260806 04:05:47.749775 3530 raft_consensus.cc:2781] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:05:47.750228 3530 raft_consensus.cc:686] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:47.751006 3530 consensus_queue.cc:237] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:47.757136 3085 catalog_manager.cc:5495] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 2 to 3, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 3 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:05:47.983228 3460 raft_consensus.cc:1261] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:47.984536 3530 consensus_queue.cc:1035] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:47.993849 3238 raft_consensus.cc:1261] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:47.995292 3530 consensus_queue.cc:1035] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.001787 3352 raft_consensus.cc:1261] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.003178 3644 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.012528 3460 raft_consensus.cc:1261] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.014247 3646 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:05:48.043064 3460 raft_consensus.cc:1261] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.044564 3666 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.053191 3238 raft_consensus.cc:1261] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.054837 3530 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.068642 3352 raft_consensus.cc:1261] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.070057 3646 consensus_queue.cc:1035] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.080180 3460 raft_consensus.cc:1261] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.082525 3692 consensus_queue.cc:1035] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:05:48.165536 3238 raft_consensus.cc:1261] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:05:48.167196 3460 raft_consensus.cc:1261] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:05:48.167088 3530 consensus_queue.cc:1035] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.168776 3530 consensus_queue.cc:1035] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.184216 3352 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:05:48.186079 3352 raft_consensus.cc:1261] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.186220 3692 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:05:48.189065 3692 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:05:48.207032 3460 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:05:48.207731 3454 raft_consensus.cc:1261] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:05:48.210531 3692 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:48.211843 3646 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:05:50.496248 3041 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260806 04:05:50.497129 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:50.520279 3239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:51648
I20260806 04:05:50.520797 3239 raft_consensus.cc:595] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Received request to transfer leadership to TS 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:50.533017 3354 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:53182
I20260806 04:05:50.533545 3354 raft_consensus.cc:595] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Received request to transfer leadership to TS 8983098d2f3843199e31fe0c278e12a3
I20260806 04:05:50.534785 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:05:50.535445 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:50.618983 3689 raft_consensus.cc:979] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:05:50.619633 3702 raft_consensus.cc:1067] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Signalling peer 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:05:50.621620 3454 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797"
dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:35636
I20260806 04:05:50.622411 3454 raft_consensus.cc:482] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:50.622982 3454 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:50.626238 3692 raft_consensus.cc:979] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:05:50.626799 3707 raft_consensus.cc:1067] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Signalling peer 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:05:50.628628 3460 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3"
dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:35644
I20260806 04:05:50.629056 3460 raft_consensus.cc:482] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:05:50.629304 3460 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:50.630067 3454 raft_consensus.cc:504] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:50.633117 3454 leader_election.cc:288] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 3 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:50.633656 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:50.633769 3354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:50.634334 3239 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:50.634433 3354 raft_consensus.cc:3032] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:05:50.635022 3354 raft_consensus.cc:729] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:05:50.634503 3460 raft_consensus.cc:504] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:50.635972 3354 consensus_queue.cc:260] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:50.638232 3354 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:50.639135 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:05:50.638106 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:05:50.639909 3352 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:50.640057 3238 raft_consensus.cc:3032] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:05:50.640599 3238 raft_consensus.cc:729] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:05:50.641518 3238 consensus_queue.cc:260] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:50.637125 3460 leader_election.cc:288] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 3 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:05:50.643805 3238 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:05:50.643043 3239 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 3.
I20260806 04:05:50.646211 3404 leader_election.cc:302] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:50.647128 3354 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 3.
I20260806 04:05:50.647277 3643 raft_consensus.cc:2781] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:05:50.653996 3643 raft_consensus.cc:686] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Becoming Leader. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:05:50.654909 3643 consensus_queue.cc:237] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:50.656484 3238 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 3.
I20260806 04:05:50.657562 3404 leader_election.cc:302] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:05:50.658586 3352 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 3.
I20260806 04:05:50.660386 3709 raft_consensus.cc:2781] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:05:50.661005 3709 raft_consensus.cc:686] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Becoming Leader. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:05:50.662029 3709 consensus_queue.cc:237] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:05:50.671080 3085 catalog_manager.cc:5495] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 reported cstate change: term changed from 2 to 3, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67). New cstate: current_term: 3 leader_uuid: "8983098d2f3843199e31fe0c278e12a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: HEALTHY } } }
I20260806 04:05:50.686594 3085 catalog_manager.cc:5495] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 reported cstate change: term changed from 2 to 3, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67). New cstate: current_term: 3 leader_uuid: "8983098d2f3843199e31fe0c278e12a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: HEALTHY } } }
I20260806 04:05:51.045631 3354 raft_consensus.cc:1261] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:05:51.046999 3714 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:05:51.056397 3238 raft_consensus.cc:1261] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:05:51.058379 3643 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:05:51.172726 3352 raft_consensus.cc:1261] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:05:51.173956 3643 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:05:51.182461 3238 raft_consensus.cc:1261] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:05:51.184190 3714 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:05:51.536317 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:51.541917 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:51.542352 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:52.543551 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:52.548756 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:52.549222 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:53.550115 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:53.554778 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:53.555254 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:54.556155 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:54.561020 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:54.561519 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:55.562427 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:55.567132 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:55.567585 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:56.568518 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:56.574609 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:56.575098 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:57.575969 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:57.582062 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:57.582587 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:58.583519 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:58.588876 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:58.589368 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:05:59.590304 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:05:59.595345 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:05:59.595722 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:00.624740 3041 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260806 04:06:00.629506 3238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:44414
I20260806 04:06:00.630290 3238 raft_consensus.cc:595] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:06:00.635743 3239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9323adac1819491f8d688196ca1a696a"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:44414
I20260806 04:06:00.636410 3239 raft_consensus.cc:595] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:06:00.641330 3239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:44414
I20260806 04:06:00.641875 3239 raft_consensus.cc:595] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:06:00.660012 3454 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3"
dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:40516
I20260806 04:06:00.660609 3454 raft_consensus.cc:595] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:06:00.667934 3454 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797"
dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
mode: GRACEFUL
new_leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:40516
I20260806 04:06:00.668510 3454 raft_consensus.cc:595] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Received request to transfer leadership to TS 9040ee4726534f72b52b4c4fd0188fc3
I20260806 04:06:00.673439 3754 raft_consensus.cc:979] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.674113 3750 raft_consensus.cc:1067] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.675809 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57864
I20260806 04:06:00.676323 3352 raft_consensus.cc:482] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:00.676738 3352 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.682010 3352 raft_consensus.cc:504] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.684183 3352 leader_election.cc:288] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 4 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:00.685454 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:00.686316 3239 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.685457 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd58b8e6c32b4b1ba07225530f879797" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:00.687268 3454 raft_consensus.cc:3032] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:00.687654 3454 raft_consensus.cc:729] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:06:00.688450 3454 consensus_queue.cc:260] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.689904 3454 raft_consensus.cc:3037] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.693857 3239 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 4.
I20260806 04:06:00.695307 3300 leader_election.cc:302] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:06:00.696187 3744 raft_consensus.cc:2781] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:00.696556 3454 raft_consensus.cc:2454] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 4.
I20260806 04:06:00.697346 3744 raft_consensus.cc:686] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:00.698225 3744 consensus_queue.cc:237] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.705575 3085 catalog_manager.cc:5495] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 3 to 4, leader changed from 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 4 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:00.729933 3754 raft_consensus.cc:979] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.730460 3750 raft_consensus.cc:1067] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.731902 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57864
I20260806 04:06:00.732401 3352 raft_consensus.cc:482] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:00.732667 3352 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.738044 3352 raft_consensus.cc:504] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.739912 3352 leader_election.cc:288] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 4 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:00.740688 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:00.741302 3239 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.742731 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b09fbea2f484bf28173d95655b8a0a3" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:00.743366 3460 raft_consensus.cc:3032] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:00.743659 3460 raft_consensus.cc:729] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:06:00.744240 3460 consensus_queue.cc:260] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.745239 3460 raft_consensus.cc:3037] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.747062 3239 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 4.
I20260806 04:06:00.748135 3300 leader_election.cc:302] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:06:00.748940 3744 raft_consensus.cc:2781] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:00.749382 3744 raft_consensus.cc:686] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:00.750291 3460 raft_consensus.cc:2454] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 4.
I20260806 04:06:00.750133 3744 consensus_queue.cc:237] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.757272 3085 catalog_manager.cc:5495] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 3 to 4, leader changed from 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 4 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:00.816939 3766 raft_consensus.cc:979] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.817514 3692 raft_consensus.cc:1067] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.818938 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57860
I20260806 04:06:00.819391 3352 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:00.819613 3352 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.823928 3352 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:06:00.825378 3352 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 4 election: Requested vote from peers 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789), 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:06:00.826486 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:00.826484 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:00.827262 3460 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.827260 3239 raft_consensus.cc:3032] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:00.827817 3239 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:06:00.828640 3239 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:06:00.830057 3239 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:00.834584 3460 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 4.
I20260806 04:06:00.836151 3299 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:06:00.836987 3239 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 4.
I20260806 04:06:00.837339 3755 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:00.837949 3755 raft_consensus.cc:686] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:00.838935 3755 consensus_queue.cc:237] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:06:00.846742 3085 catalog_manager.cc:5495] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 3 to 4, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 4 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:00.909099 3766 raft_consensus.cc:979] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.909593 3692 raft_consensus.cc:1067] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:00.911091 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9323adac1819491f8d688196ca1a696a"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57860
I20260806 04:06:00.911554 3352 raft_consensus.cc:482] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:00.911844 3352 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:00.916569 3352 raft_consensus.cc:504] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.917982 3352 leader_election.cc:288] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 3 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:00.919008 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:00.919008 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9323adac1819491f8d688196ca1a696a" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:00.919773 3460 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:00.919883 3239 raft_consensus.cc:3032] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:00.920238 3239 raft_consensus.cc:729] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:06:00.920897 3239 consensus_queue.cc:260] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.921901 3239 raft_consensus.cc:3037] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:00.924398 3460 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 3.
I20260806 04:06:00.925453 3299 leader_election.cc:302] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:06:00.926201 3755 raft_consensus.cc:2781] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:00.926607 3755 raft_consensus.cc:686] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:00.927141 3239 raft_consensus.cc:2454] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 3.
I20260806 04:06:00.927590 3755 consensus_queue.cc:237] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:00.934402 3085 catalog_manager.cc:5495] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 2 to 3, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 3 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:01.078819 3460 raft_consensus.cc:1261] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:01.080080 3755 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:01.091334 3239 raft_consensus.cc:1261] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:01.092896 3744 consensus_queue.cc:1035] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:01.157491 3766 raft_consensus.cc:979] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: : Instructing follower 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:01.157975 3692 raft_consensus.cc:1067] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Signalling peer 9040ee4726534f72b52b4c4fd0188fc3 to start an election
I20260806 04:06:01.159430 3352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
from {username='slave'} at 127.0.0.1:57860
I20260806 04:06:01.159983 3352 raft_consensus.cc:482] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:01.160383 3352 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:01.165236 3352 raft_consensus.cc:504] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:01.167305 3352 leader_election.cc:288] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 3 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:01.167945 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:01.168293 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "9040ee4726534f72b52b4c4fd0188fc3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:01.168529 3460 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:01.168892 3239 raft_consensus.cc:3032] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:01.169234 3239 raft_consensus.cc:729] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:06:01.170166 3239 consensus_queue.cc:260] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:01.171262 3239 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:01.173349 3460 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 3.
I20260806 04:06:01.174301 3299 leader_election.cc:302] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983098d2f3843199e31fe0c278e12a3, 9040ee4726534f72b52b4c4fd0188fc3; no voters:
I20260806 04:06:01.175052 3744 raft_consensus.cc:2781] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:01.175561 3744 raft_consensus.cc:686] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Becoming Leader. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:01.176445 3744 consensus_queue.cc:237] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:01.176936 3239 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9040ee4726534f72b52b4c4fd0188fc3 in term 3.
I20260806 04:06:01.183914 3087 catalog_manager.cc:5495] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 reported cstate change: term changed from 2 to 3, leader changed from 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65) to 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66). New cstate: current_term: 3 leader_uuid: "9040ee4726534f72b52b4c4fd0188fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:01.220626 3239 raft_consensus.cc:1261] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:01.222223 3755 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:01.223517 3460 raft_consensus.cc:1261] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:01.224797 3755 consensus_queue.cc:1035] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:01.405407 3239 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:06:01.406677 3755 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:01.415844 3460 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:06:01.417394 3755 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:01.487396 3239 raft_consensus.cc:1261] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:01.488787 3755 consensus_queue.cc:1035] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:01.501037 3460 raft_consensus.cc:1261] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:01.502527 3782 consensus_queue.cc:1035] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:01.670437 3460 raft_consensus.cc:1261] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:01.672421 3782 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:01.686771 3239 raft_consensus.cc:1261] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Refusing update from remote peer 9040ee4726534f72b52b4c4fd0188fc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:01.690097 3755 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:03.705142 3041 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260806 04:06:03.705847 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:03.712489 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:53182
I20260806 04:06:03.713289 3352 raft_consensus.cc:595] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Received request to transfer leadership to TS 8983098d2f3843199e31fe0c278e12a3
I20260806 04:06:03.716169 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18319d34a9154a27946dda911923380a"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:53182
I20260806 04:06:03.716635 3352 raft_consensus.cc:595] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:06:03.718632 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:53182
I20260806 04:06:03.719081 3352 raft_consensus.cc:595] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:06:03.721076 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:53182
I20260806 04:06:03.721513 3352 raft_consensus.cc:595] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Received request to transfer leadership to TS 8983098d2f3843199e31fe0c278e12a3
I20260806 04:06:03.723490 3352 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f"
dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
mode: GRACEFUL
new_leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:53182
I20260806 04:06:03.723906 3352 raft_consensus.cc:595] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Received request to transfer leadership to TS 2d3a226c500b4528a2dea7cd30910d6a
I20260806 04:06:03.724824 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:06:03.725273 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:03.782228 3793 raft_consensus.cc:979] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:06:03.782958 3744 raft_consensus.cc:1067] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Signalling peer 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:06:03.784289 3460 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78"
dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:35636
I20260806 04:06:03.784758 3460 raft_consensus.cc:482] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:03.785050 3460 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:03.790159 3460 raft_consensus.cc:504] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:06:03.792094 3460 leader_election.cc:288] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 5 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817)
I20260806 04:06:03.792977 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:06:03.793202 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749844bfd19e4bdb8bd8aad14b132a78" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:03.793540 3352 raft_consensus.cc:3032] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:06:03.793732 3239 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:03.793871 3352 raft_consensus.cc:729] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:03.794739 3352 consensus_queue.cc:260] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:06:03.795706 3352 raft_consensus.cc:3037] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:03.798516 3239 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 5.
I20260806 04:06:03.799693 3404 leader_election.cc:302] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:06:03.800372 3352 raft_consensus.cc:2454] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 5.
I20260806 04:06:03.800513 3750 raft_consensus.cc:2781] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:06:03.801637 3750 raft_consensus.cc:686] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 5 LEADER]: Becoming Leader. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:06:03.802549 3750 consensus_queue.cc:237] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } }
I20260806 04:06:03.809577 3085 catalog_manager.cc:5495] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 reported cstate change: term changed from 4 to 5, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67). New cstate: current_term: 5 leader_uuid: "8983098d2f3843199e31fe0c278e12a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:03.988628 3793 raft_consensus.cc:979] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:06:03.989099 3744 raft_consensus.cc:1067] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:06:03.990430 3239 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:06:03.990931 3239 raft_consensus.cc:482] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:03.991199 3239 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:03.995965 3239 raft_consensus.cc:504] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:03.997913 3239 leader_election.cc:288] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 4 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:03.998876 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:06:03.999452 3352 raft_consensus.cc:3032] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:03.999312 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40f9b110c2e4169b055a1c0de83d19f" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:03.999740 3352 raft_consensus.cc:729] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:03.999907 3460 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:04.000520 3352 consensus_queue.cc:260] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.001780 3352 raft_consensus.cc:3037] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:04.004884 3460 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 4.
I20260806 04:06:04.006063 3190 leader_election.cc:302] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:06:04.006177 3352 raft_consensus.cc:2454] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 4.
I20260806 04:06:04.007731 3801 raft_consensus.cc:2781] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:04.008749 3801 raft_consensus.cc:686] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 4 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:06:04.009583 3801 consensus_queue.cc:237] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.016266 3085 catalog_manager.cc:5495] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 3 to 4, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 4 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:04.080422 3744 raft_consensus.cc:979] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:06:04.080901 3793 raft_consensus.cc:1067] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:06:04.082372 3239 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:06:04.082851 3239 raft_consensus.cc:482] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:04.083081 3239 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:04.087332 3239 raft_consensus.cc:504] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.088701 3239 leader_election.cc:288] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 4 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:04.089711 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:04.089766 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ab0cb20398435ebc28a993a65026b6" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:06:04.090301 3460 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:04.090374 3352 raft_consensus.cc:3032] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:04.090762 3352 raft_consensus.cc:729] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:04.091406 3352 consensus_queue.cc:260] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.092275 3352 raft_consensus.cc:3037] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:04.095055 3460 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 4.
I20260806 04:06:04.098878 3352 raft_consensus.cc:2454] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 4.
I20260806 04:06:04.099735 3190 leader_election.cc:302] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:06:04.109503 3801 raft_consensus.cc:2781] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:04.110266 3801 raft_consensus.cc:686] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:06:04.111200 3801 consensus_queue.cc:237] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.120270 3744 raft_consensus.cc:979] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:06:04.120790 3793 raft_consensus.cc:1067] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Signalling peer 8983098d2f3843199e31fe0c278e12a3 to start an election
I20260806 04:06:04.121206 3085 catalog_manager.cc:5495] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 3 to 4, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 4 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:04.123322 3460 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6"
dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
from {username='slave'} at 127.0.0.1:35636
I20260806 04:06:04.123983 3460 raft_consensus.cc:482] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:04.124408 3460 raft_consensus.cc:3037] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:04.132296 3460 raft_consensus.cc:504] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.134925 3460 leader_election.cc:288] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 2 election: Requested vote from peers 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817), 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521)
I20260806 04:06:04.135308 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
I20260806 04:06:04.135500 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ab8d40e1ba45ad9549a392cbdd87b6" candidate_uuid: "8983098d2f3843199e31fe0c278e12a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:06:04.135941 3239 raft_consensus.cc:3037] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:04.136090 3352 raft_consensus.cc:3032] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:04.136482 3352 raft_consensus.cc:729] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:04.137277 3352 consensus_queue.cc:260] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.138819 3352 raft_consensus.cc:3037] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:04.142853 3239 raft_consensus.cc:2454] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 2.
I20260806 04:06:04.144119 3404 leader_election.cc:302] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:06:04.145455 3750 raft_consensus.cc:2781] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:04.145931 3750 raft_consensus.cc:686] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 LEADER]: Becoming Leader. State: Replica: 8983098d2f3843199e31fe0c278e12a3, State: Running, Role: LEADER
I20260806 04:06:04.146886 3750 consensus_queue.cc:237] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.148473 3352 raft_consensus.cc:2454] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8983098d2f3843199e31fe0c278e12a3 in term 2.
I20260806 04:06:04.156216 3085 catalog_manager.cc:5495] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 reported cstate change: term changed from 1 to 2, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "8983098d2f3843199e31fe0c278e12a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:04.245620 3744 raft_consensus.cc:979] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: : Instructing follower 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:06:04.246085 3793 raft_consensus.cc:1067] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Signalling peer 2d3a226c500b4528a2dea7cd30910d6a to start an election
I20260806 04:06:04.247670 3239 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18319d34a9154a27946dda911923380a"
dest_uuid: "2d3a226c500b4528a2dea7cd30910d6a"
from {username='slave'} at 127.0.0.1:44404
I20260806 04:06:04.248091 3239 raft_consensus.cc:482] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:04.248369 3239 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:04.252660 3239 raft_consensus.cc:504] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.253553 3352 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 5 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:06:04.254109 3239 leader_election.cc:288] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 3 election: Requested vote from peers 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521), 8983098d2f3843199e31fe0c278e12a3 (127.2.248.67:42789)
I20260806 04:06:04.255024 3352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9040ee4726534f72b52b4c4fd0188fc3"
I20260806 04:06:04.255112 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18319d34a9154a27946dda911923380a" candidate_uuid: "2d3a226c500b4528a2dea7cd30910d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8983098d2f3843199e31fe0c278e12a3"
I20260806 04:06:04.254722 3750 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:04.255673 3352 raft_consensus.cc:3032] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:04.255807 3460 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:04.255998 3352 raft_consensus.cc:729] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9040ee4726534f72b52b4c4fd0188fc3, State: Running, Role: LEADER
I20260806 04:06:04.256753 3352 consensus_queue.cc:260] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.257840 3352 raft_consensus.cc:3037] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:04.262241 3460 raft_consensus.cc:2454] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 3.
I20260806 04:06:04.263284 3190 leader_election.cc:302] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3a226c500b4528a2dea7cd30910d6a, 8983098d2f3843199e31fe0c278e12a3; no voters:
I20260806 04:06:04.265056 3801 raft_consensus.cc:2781] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:04.265556 3801 raft_consensus.cc:686] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Becoming Leader. State: Replica: 2d3a226c500b4528a2dea7cd30910d6a, State: Running, Role: LEADER
I20260806 04:06:04.266502 3801 consensus_queue.cc:237] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } }
I20260806 04:06:04.272928 3239 raft_consensus.cc:1261] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 5 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:06:04.274999 3794 consensus_queue.cc:1035] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:04.277321 3352 raft_consensus.cc:2454] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d3a226c500b4528a2dea7cd30910d6a in term 3.
I20260806 04:06:04.301715 3087 catalog_manager.cc:5495] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a reported cstate change: term changed from 2 to 3, leader changed from 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66) to 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "2d3a226c500b4528a2dea7cd30910d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:04.549467 3460 raft_consensus.cc:1261] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:06:04.550587 3801 consensus_queue.cc:1035] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:04.560328 3354 raft_consensus.cc:1261] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:06:04.561951 3801 consensus_queue.cc:1035] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:04.588181 3460 raft_consensus.cc:1261] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:04.589354 3801 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:04.597852 3354 raft_consensus.cc:1261] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:04.600970 3802 consensus_queue.cc:1035] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:04.676901 3238 raft_consensus.cc:1261] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:04.678278 3811 consensus_queue.cc:1035] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3a226c500b4528a2dea7cd30910d6a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:04.687206 3354 raft_consensus.cc:1261] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Refusing update from remote peer 8983098d2f3843199e31fe0c278e12a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:04.689112 3811 consensus_queue.cc:1035] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:04.718130 3354 raft_consensus.cc:1261] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:04.719533 3802 consensus_queue.cc:1035] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9040ee4726534f72b52b4c4fd0188fc3" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:04.726187 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:04.728199 3460 raft_consensus.cc:1261] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Refusing update from remote peer 2d3a226c500b4528a2dea7cd30910d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:04.730235 3821 consensus_queue.cc:1035] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983098d2f3843199e31fe0c278e12a3" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:04.734335 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:04.735248 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:05.736380 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:05.742614 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:05.743216 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:06.744102 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:06.749277 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:06.749771 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:07.750622 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:07.755551 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:07.756012 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:08.756886 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:08.762311 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:08.762806 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:09.766137 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:09.785763 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:09.788472 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:10.790050 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:10.795853 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:10.796288 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:11.797303 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:11.803261 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:11.803704 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:12.804545 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:12.810009 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:12.810529 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:13.841245 3041 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260806 04:06:13.842047 3041 tablet_server.cc:178] TabletServer@127.2.248.65:0 shutting down...
W20260806 04:06:13.852244 3300 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260806 04:06:13.858537 3300 consensus_peers.cc:487] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 -> Peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Couldn't send request to peer 2d3a226c500b4528a2dea7cd30910d6a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:13.886530 3041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:06:13.887627 3041 tablet_replica.cc:330] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
W20260806 04:06:13.888259 3404 consensus_peers.cc:487] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 -> Peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Couldn't send request to peer 2d3a226c500b4528a2dea7cd30910d6a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:13.888665 3041 raft_consensus.cc:2229] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:06:13.890007 3041 raft_consensus.cc:2258] T 55ab0cb20398435ebc28a993a65026b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.892813 3041 tablet_replica.cc:330] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.893399 3041 raft_consensus.cc:2229] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:13.893924 3041 raft_consensus.cc:2258] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 2d3a226c500b4528a2dea7cd30910d6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.895915 3041 tablet_replica.cc:330] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.896467 3041 raft_consensus.cc:2229] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:13.896920 3041 raft_consensus.cc:2258] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 2d3a226c500b4528a2dea7cd30910d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.898875 3041 tablet_replica.cc:330] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.899411 3041 raft_consensus.cc:2229] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:13.899911 3041 raft_consensus.cc:2258] T dd58b8e6c32b4b1ba07225530f879797 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.901644 3041 tablet_replica.cc:330] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.902215 3041 raft_consensus.cc:2229] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 LEADER]: Raft consensus shutting down.
I20260806 04:06:13.903136 3041 raft_consensus.cc:2258] T 18319d34a9154a27946dda911923380a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.905041 3041 tablet_replica.cc:330] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.905601 3041 raft_consensus.cc:2229] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:13.906056 3041 raft_consensus.cc:2258] T 9323adac1819491f8d688196ca1a696a P 2d3a226c500b4528a2dea7cd30910d6a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.908030 3041 tablet_replica.cc:330] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.908565 3041 raft_consensus.cc:2229] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:13.909070 3041 raft_consensus.cc:2258] T 6b09fbea2f484bf28173d95655b8a0a3 P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.910845 3041 tablet_replica.cc:330] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.911381 3041 raft_consensus.cc:2229] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:13.911875 3041 raft_consensus.cc:2258] T 749844bfd19e4bdb8bd8aad14b132a78 P 2d3a226c500b4528a2dea7cd30910d6a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:13.913964 3041 tablet_replica.cc:330] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a: stopping tablet replica
I20260806 04:06:13.914505 3041 raft_consensus.cc:2229] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:06:13.915428 3041 raft_consensus.cc:2258] T e40f9b110c2e4169b055a1c0de83d19f P 2d3a226c500b4528a2dea7cd30910d6a [term 4 FOLLOWER]: Raft consensus is shut down!
W20260806 04:06:14.038491 3404 consensus_peers.cc:487] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 -> Peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Couldn't send request to peer 2d3a226c500b4528a2dea7cd30910d6a. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.65:46817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260806 04:06:14.038757 3300 consensus_peers.cc:487] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 -> Peer 2d3a226c500b4528a2dea7cd30910d6a (127.2.248.65:46817): Couldn't send request to peer 2d3a226c500b4528a2dea7cd30910d6a. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.65:46817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:14.039872 3041 tablet_server.cc:194] TabletServer@127.2.248.65:0 shutdown complete.
I20260806 04:06:14.060910 3041 tablet_server.cc:178] TabletServer@127.2.248.66:0 shutting down...
I20260806 04:06:14.089851 3041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:06:14.090528 3041 tablet_replica.cc:330] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.091614 3041 raft_consensus.cc:2229] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.092326 3041 raft_consensus.cc:2258] T e40f9b110c2e4169b055a1c0de83d19f P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.094254 3041 tablet_replica.cc:330] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.094822 3041 raft_consensus.cc:2229] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.095934 3041 raft_consensus.cc:2258] T 9323adac1819491f8d688196ca1a696a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.097736 3041 tablet_replica.cc:330] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.098268 3041 raft_consensus.cc:2229] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.098778 3041 raft_consensus.cc:2258] T 18319d34a9154a27946dda911923380a P 9040ee4726534f72b52b4c4fd0188fc3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.100755 3041 tablet_replica.cc:330] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.101403 3041 raft_consensus.cc:2229] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.102594 3041 raft_consensus.cc:2258] T dd58b8e6c32b4b1ba07225530f879797 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.105111 3041 tablet_replica.cc:330] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.105795 3041 raft_consensus.cc:2229] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.106254 3041 raft_consensus.cc:2258] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.108144 3041 tablet_replica.cc:330] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.108623 3041 raft_consensus.cc:2229] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.109025 3041 raft_consensus.cc:2258] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 9040ee4726534f72b52b4c4fd0188fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.110622 3041 tablet_replica.cc:330] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.111157 3041 raft_consensus.cc:2229] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.111948 3041 raft_consensus.cc:2258] T 6b09fbea2f484bf28173d95655b8a0a3 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.113525 3041 tablet_replica.cc:330] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.114007 3041 raft_consensus.cc:2229] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.114423 3041 raft_consensus.cc:2258] T 749844bfd19e4bdb8bd8aad14b132a78 P 9040ee4726534f72b52b4c4fd0188fc3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.116077 3041 tablet_replica.cc:330] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3: stopping tablet replica
I20260806 04:06:14.116518 3041 raft_consensus.cc:2229] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.116921 3041 raft_consensus.cc:2258] T 55ab0cb20398435ebc28a993a65026b6 P 9040ee4726534f72b52b4c4fd0188fc3 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260806 04:06:14.127616 3403 consensus_peers.cc:487] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 -> Peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Couldn't send request to peer 9040ee4726534f72b52b4c4fd0188fc3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:14.146766 3041 tablet_server.cc:194] TabletServer@127.2.248.66:0 shutdown complete.
W20260806 04:06:14.157039 3403 consensus_peers.cc:487] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 -> Peer 9040ee4726534f72b52b4c4fd0188fc3 (127.2.248.66:45521): Couldn't send request to peer 9040ee4726534f72b52b4c4fd0188fc3. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.66:45521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:14.168242 3041 tablet_server.cc:178] TabletServer@127.2.248.67:0 shutting down...
I20260806 04:06:14.201033 3041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:06:14.201730 3041 tablet_replica.cc:330] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.202409 3041 raft_consensus.cc:2229] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.202997 3041 raft_consensus.cc:2258] T dd58b8e6c32b4b1ba07225530f879797 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.204993 3041 tablet_replica.cc:330] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.205534 3041 raft_consensus.cc:2229] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.206079 3041 raft_consensus.cc:2258] T 9323adac1819491f8d688196ca1a696a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.207937 3041 tablet_replica.cc:330] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.208447 3041 raft_consensus.cc:2229] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.208973 3041 raft_consensus.cc:2258] T 18319d34a9154a27946dda911923380a P 8983098d2f3843199e31fe0c278e12a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.210589 3041 tablet_replica.cc:330] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.211161 3041 raft_consensus.cc:2229] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 5 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.212004 3041 raft_consensus.cc:2258] T 749844bfd19e4bdb8bd8aad14b132a78 P 8983098d2f3843199e31fe0c278e12a3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.213976 3041 tablet_replica.cc:330] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.214461 3041 raft_consensus.cc:2229] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.214936 3041 raft_consensus.cc:2258] T 6b09fbea2f484bf28173d95655b8a0a3 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.216603 3041 tablet_replica.cc:330] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.217087 3041 raft_consensus.cc:2229] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.217872 3041 raft_consensus.cc:2258] T 9c6c24b5a0ce4e7c9941928a3a518f05 P 8983098d2f3843199e31fe0c278e12a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.219607 3041 tablet_replica.cc:330] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.220114 3041 raft_consensus.cc:2229] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.220897 3041 raft_consensus.cc:2258] T 13ab8d40e1ba45ad9549a392cbdd87b6 P 8983098d2f3843199e31fe0c278e12a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.222576 3041 tablet_replica.cc:330] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.223111 3041 raft_consensus.cc:2229] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.223558 3041 raft_consensus.cc:2258] T e40f9b110c2e4169b055a1c0de83d19f P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.225208 3041 tablet_replica.cc:330] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3: stopping tablet replica
I20260806 04:06:14.225656 3041 raft_consensus.cc:2229] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:14.226104 3041 raft_consensus.cc:2258] T 55ab0cb20398435ebc28a993a65026b6 P 8983098d2f3843199e31fe0c278e12a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.254426 3041 tablet_server.cc:194] TabletServer@127.2.248.67:0 shutdown complete.
I20260806 04:06:14.273043 3041 master.cc:537] Master@127.2.248.126:34201 shutting down...
I20260806 04:06:14.302610 3174 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260806 04:06:14.561362 3041 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:06:14.562002 3041 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:14.562424 3041 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6c3a31e75e6349c0844e2b3d6369efc1: stopping tablet replica
I20260806 04:06:14.582235 3041 master.cc:552] Master@127.2.248.126:34201 shutdown complete.
2d3a226c500b4528a2dea7cd30910d6a 4
8983098d2f3843199e31fe0c278e12a3 1
9040ee4726534f72b52b4c4fd0188fc3 4
2d3a226c500b4528a2dea7cd30910d6a 3
8983098d2f3843199e31fe0c278e12a3 3
9040ee4726534f72b52b4c4fd0188fc3 3
2d3a226c500b4528a2dea7cd30910d6a 0
8983098d2f3843199e31fe0c278e12a3 1
9040ee4726534f72b52b4c4fd0188fc3 8
2d3a226c500b4528a2dea7cd30910d6a 3
8983098d2f3843199e31fe0c278e12a3 3
9040ee4726534f72b52b4c4fd0188fc3 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (33093 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260806 04:06:14.628857 3041 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.248.126:33591
I20260806 04:06:14.630053 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:14.636123 3870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:14.636438 3872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:14.637441 3874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:14.638710 3041 server_base.cc:1034] running on GCE node
I20260806 04:06:14.639885 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:14.640069 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:14.640259 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989174640229 us; error 0 us; skew 500 ppm
I20260806 04:06:14.640808 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:14.646214 3041 webserver.cc:460] Webserver started at http://127.2.248.126:36923/ using document root <none> and password file <none>
I20260806 04:06:14.647156 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:14.647368 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:14.647671 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:14.648993 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/master-0-root/instance:
uuid: "c2b6d374c8d8472c8fd3bb0253ca3508"
format_stamp: "Formatted at 2026-08-06 04:06:14 on dist-test-slave-3kk6"
I20260806 04:06:14.653752 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:06:14.657132 3879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:14.657915 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:06:14.658201 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/master-0-root
uuid: "c2b6d374c8d8472c8fd3bb0253ca3508"
format_stamp: "Formatted at 2026-08-06 04:06:14 on dist-test-slave-3kk6"
I20260806 04:06:14.658499 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:14.669519 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:14.670992 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:14.713212 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.126:33591
I20260806 04:06:14.713320 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.126:33591 every 8 connection(s)
I20260806 04:06:14.717326 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:14.729205 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508: Bootstrap starting.
I20260806 04:06:14.733906 3949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:14.738525 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508: No bootstrap required, opened a new log
I20260806 04:06:14.740679 3949 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" member_type: VOTER }
I20260806 04:06:14.741192 3949 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:14.741438 3949 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b6d374c8d8472c8fd3bb0253ca3508, State: Initialized, Role: FOLLOWER
I20260806 04:06:14.742084 3949 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" member_type: VOTER }
I20260806 04:06:14.742579 3949 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:06:14.742851 3949 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:06:14.743130 3949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:14.747130 3949 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" member_type: VOTER }
I20260806 04:06:14.747702 3949 leader_election.cc:302] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2b6d374c8d8472c8fd3bb0253ca3508; no voters:
I20260806 04:06:14.748906 3949 leader_election.cc:288] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:06:14.749274 3954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:14.750793 3954 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 1 LEADER]: Becoming Leader. State: Replica: c2b6d374c8d8472c8fd3bb0253ca3508, State: Running, Role: LEADER
I20260806 04:06:14.751430 3954 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" member_type: VOTER }
I20260806 04:06:14.752091 3949 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:06:14.756525 3955 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" member_type: VOTER } }
I20260806 04:06:14.757376 3955 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [sys.catalog]: This master's current role is: LEADER
I20260806 04:06:14.757434 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2b6d374c8d8472c8fd3bb0253ca3508. Latest consensus state: current_term: 1 leader_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b6d374c8d8472c8fd3bb0253ca3508" member_type: VOTER } }
I20260806 04:06:14.758459 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [sys.catalog]: This master's current role is: LEADER
I20260806 04:06:14.763540 3963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:06:14.771837 3963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:06:14.772908 3041 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 04:06:14.780961 3963 catalog_manager.cc:1347] Generated new cluster ID: 2b320ca35bf54f068d81ca9d93c43e1e
I20260806 04:06:14.781279 3963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:06:14.814167 3963 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:06:14.815958 3963 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:06:14.833820 3963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508: Generated new TSK 0
I20260806 04:06:14.834722 3963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:06:14.903874 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:14.910008 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:14.911422 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:14.913542 3041 server_base.cc:1034] running on GCE node
W20260806 04:06:14.913828 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:14.914901 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:14.915107 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:14.915267 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989174915255 us; error 0 us; skew 500 ppm
I20260806 04:06:14.915823 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:14.918275 3041 webserver.cc:460] Webserver started at http://127.2.248.65:43853/ using document root <none> and password file <none>
I20260806 04:06:14.918848 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:14.919054 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:14.919329 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:14.920452 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-0-root/instance:
uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
format_stamp: "Formatted at 2026-08-06 04:06:14 on dist-test-slave-3kk6"
I20260806 04:06:14.925262 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:06:14.928524 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:14.929286 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:06:14.929597 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-0-root
uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
format_stamp: "Formatted at 2026-08-06 04:06:14 on dist-test-slave-3kk6"
I20260806 04:06:14.929901 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:14.967880 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:14.969221 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:14.970846 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:14.973294 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:14.973508 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:14.973755 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:14.973942 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.015659 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.65:46573
I20260806 04:06:15.015762 4085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.65:46573 every 8 connection(s)
I20260806 04:06:15.020255 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:15.028234 4094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:15.029755 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:15.032078 3041 server_base.cc:1034] running on GCE node
W20260806 04:06:15.032579 4099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:15.034166 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:15.034389 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:15.034600 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989175034582 us; error 0 us; skew 500 ppm
I20260806 04:06:15.035315 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:15.037052 4088 heartbeater.cc:347] Connected to a master server at 127.2.248.126:33591
I20260806 04:06:15.037398 4088 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:15.038190 4088 heartbeater.cc:511] Master 127.2.248.126:33591 requested a full tablet report, sending...
I20260806 04:06:15.038231 3041 webserver.cc:460] Webserver started at http://127.2.248.66:37255/ using document root <none> and password file <none>
I20260806 04:06:15.038875 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:15.039103 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:15.039446 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:15.040616 3898 ts_manager.cc:194] Registered new tserver with Master: c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573)
I20260806 04:06:15.041051 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-1-root/instance:
uuid: "8d6e3133f3f04620ade89590b792b4ab"
format_stamp: "Formatted at 2026-08-06 04:06:15 on dist-test-slave-3kk6"
I20260806 04:06:15.043486 3898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36162
I20260806 04:06:15.047171 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260806 04:06:15.050757 4105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.051505 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:06:15.051800 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-1-root
uuid: "8d6e3133f3f04620ade89590b792b4ab"
format_stamp: "Formatted at 2026-08-06 04:06:15 on dist-test-slave-3kk6"
I20260806 04:06:15.052155 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:15.078357 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:15.079679 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:15.081188 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:15.083582 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:15.083783 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.084048 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:15.084199 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.125734 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.66:46015
I20260806 04:06:15.125862 4204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.66:46015 every 8 connection(s)
I20260806 04:06:15.130548 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:15.137547 4208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:15.139394 4209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:15.143658 4213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:15.144635 4205 heartbeater.cc:347] Connected to a master server at 127.2.248.126:33591
I20260806 04:06:15.145076 4205 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:15.145107 3041 server_base.cc:1034] running on GCE node
I20260806 04:06:15.146014 4205 heartbeater.cc:511] Master 127.2.248.126:33591 requested a full tablet report, sending...
I20260806 04:06:15.146057 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:15.146358 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:15.146529 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989175146510 us; error 0 us; skew 500 ppm
I20260806 04:06:15.147245 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:15.148150 3898 ts_manager.cc:194] Registered new tserver with Master: 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:15.149914 3041 webserver.cc:460] Webserver started at http://127.2.248.67:39057/ using document root <none> and password file <none>
I20260806 04:06:15.150107 3898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36166
I20260806 04:06:15.150492 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:15.150754 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:15.151067 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:15.152396 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-2-root/instance:
uuid: "c30b4ebfcccd4fd080c384375c5b1528"
format_stamp: "Formatted at 2026-08-06 04:06:15 on dist-test-slave-3kk6"
I20260806 04:06:15.157335 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:06:15.160745 4218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.161510 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:06:15.161792 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-2-root
uuid: "c30b4ebfcccd4fd080c384375c5b1528"
format_stamp: "Formatted at 2026-08-06 04:06:15 on dist-test-slave-3kk6"
I20260806 04:06:15.162079 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785989141369922-3041-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:15.173502 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:15.174749 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:15.176149 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:15.178246 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:15.178457 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.178732 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:15.178898 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:15.219660 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.67:41145
I20260806 04:06:15.219770 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.67:41145 every 8 connection(s)
I20260806 04:06:15.235764 4320 heartbeater.cc:347] Connected to a master server at 127.2.248.126:33591
I20260806 04:06:15.236178 4320 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:15.236855 4320 heartbeater.cc:511] Master 127.2.248.126:33591 requested a full tablet report, sending...
I20260806 04:06:15.239300 3898 ts_manager.cc:194] Registered new tserver with Master: c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.239691 3041 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016734511s
I20260806 04:06:15.240996 3898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36174
I20260806 04:06:15.241706 3041 test_util.cc:268] Using random seed: 39644119
I20260806 04:06:15.267529 3898 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36182:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:06:15.270632 3898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:06:15.355350 4271 tablet_service.cc:1467] Processing CreateTablet for tablet 5e64f9fe31af460aab651551bcce4915 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 04:06:15.356052 4266 tablet_service.cc:1467] Processing CreateTablet for tablet 24140f74cbd74542aa67b1d5ac7d0588 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 04:06:15.356899 4271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e64f9fe31af460aab651551bcce4915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.357604 4266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24140f74cbd74542aa67b1d5ac7d0588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.359030 4270 tablet_service.cc:1467] Processing CreateTablet for tablet 8d766a61fb5e47059f39a048dd05cbfd (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 04:06:15.360333 4270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d766a61fb5e47059f39a048dd05cbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.363811 4269 tablet_service.cc:1467] Processing CreateTablet for tablet 768c842e242e4b758b52d8f70bd6a3e3 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 04:06:15.365087 4269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768c842e242e4b758b52d8f70bd6a3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.366364 4032 tablet_service.cc:1467] Processing CreateTablet for tablet 5e64f9fe31af460aab651551bcce4915 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 04:06:15.367739 4032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e64f9fe31af460aab651551bcce4915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.367596 4267 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf6122c7b2a443b9c8026e303cab731 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 04:06:15.368522 4267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cf6122c7b2a443b9c8026e303cab731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.369315 4019 tablet_service.cc:1467] Processing CreateTablet for tablet 9e09dc4c75a94810ac73103b0b9fe4dc (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 04:06:15.372033 4030 tablet_service.cc:1467] Processing CreateTablet for tablet 8d766a61fb5e47059f39a048dd05cbfd (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 04:06:15.372044 4265 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb6388c650c430c89eee0ee05eae6d1 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 04:06:15.372709 4263 tablet_service.cc:1467] Processing CreateTablet for tablet 81cc94dc4ae64fdbb3a48e0ff9d4b519 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 04:06:15.373644 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb6388c650c430c89eee0ee05eae6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.374398 4028 tablet_service.cc:1467] Processing CreateTablet for tablet 768c842e242e4b758b52d8f70bd6a3e3 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 04:06:15.375387 4261 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7265c5ca1448adafab545d07ffef39 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 04:06:15.376724 4261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7265c5ca1448adafab545d07ffef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.376655 4027 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf6122c7b2a443b9c8026e303cab731 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 04:06:15.379066 4025 tablet_service.cc:1467] Processing CreateTablet for tablet 24140f74cbd74542aa67b1d5ac7d0588 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 04:06:15.380755 4260 tablet_service.cc:1467] Processing CreateTablet for tablet 9e09dc4c75a94810ac73103b0b9fe4dc (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 04:06:15.382112 4260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e09dc4c75a94810ac73103b0b9fe4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.382158 4024 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb6388c650c430c89eee0ee05eae6d1 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 04:06:15.383555 4263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cc94dc4ae64fdbb3a48e0ff9d4b519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.385272 4022 tablet_service.cc:1467] Processing CreateTablet for tablet 81cc94dc4ae64fdbb3a48e0ff9d4b519 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 04:06:15.386436 4342 tablet_bootstrap.cc:492] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.387691 4021 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7265c5ca1448adafab545d07ffef39 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 04:06:15.390168 4016 tablet_service.cc:1467] Processing CreateTablet for tablet 5c188bfc44d644759a415548e24d6bf0 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 04:06:15.393081 4342 tablet_bootstrap.cc:654] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.393121 4019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e09dc4c75a94810ac73103b0b9fe4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.393918 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cf6122c7b2a443b9c8026e303cab731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.399396 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7265c5ca1448adafab545d07ffef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.401973 4028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768c842e242e4b758b52d8f70bd6a3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.407114 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d766a61fb5e47059f39a048dd05cbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.409781 4016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c188bfc44d644759a415548e24d6bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.410439 4024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb6388c650c430c89eee0ee05eae6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.411144 4022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cc94dc4ae64fdbb3a48e0ff9d4b519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.414630 4025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24140f74cbd74542aa67b1d5ac7d0588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.422360 4266 tablet_service.cc:1467] Processing CreateTablet for tablet 5c188bfc44d644759a415548e24d6bf0 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 04:06:15.423707 4266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c188bfc44d644759a415548e24d6bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.444185 4148 tablet_service.cc:1467] Processing CreateTablet for tablet 5e64f9fe31af460aab651551bcce4915 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 04:06:15.445626 4148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e64f9fe31af460aab651551bcce4915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.447677 4142 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb6388c650c430c89eee0ee05eae6d1 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 04:06:15.447927 4147 tablet_service.cc:1467] Processing CreateTablet for tablet 8d766a61fb5e47059f39a048dd05cbfd (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 04:06:15.449070 4142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb6388c650c430c89eee0ee05eae6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.451927 4141 tablet_service.cc:1467] Processing CreateTablet for tablet 81cc94dc4ae64fdbb3a48e0ff9d4b519 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 04:06:15.451968 4144 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf6122c7b2a443b9c8026e303cab731 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 04:06:15.453384 4141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cc94dc4ae64fdbb3a48e0ff9d4b519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.454072 4143 tablet_service.cc:1467] Processing CreateTablet for tablet 24140f74cbd74542aa67b1d5ac7d0588 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 04:06:15.450282 4146 tablet_service.cc:1467] Processing CreateTablet for tablet 768c842e242e4b758b52d8f70bd6a3e3 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 04:06:15.456705 4147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d766a61fb5e47059f39a048dd05cbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.458567 4139 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7265c5ca1448adafab545d07ffef39 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 04:06:15.459231 4143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24140f74cbd74542aa67b1d5ac7d0588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.459939 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7265c5ca1448adafab545d07ffef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.460587 4144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cf6122c7b2a443b9c8026e303cab731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.463021 4146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768c842e242e4b758b52d8f70bd6a3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.468544 4137 tablet_service.cc:1467] Processing CreateTablet for tablet 9e09dc4c75a94810ac73103b0b9fe4dc (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 04:06:15.469345 4136 tablet_service.cc:1467] Processing CreateTablet for tablet 5c188bfc44d644759a415548e24d6bf0 (DEFAULT_TABLE table=test-workload [id=ffc39880b76e45b5bf8d60a6342f6174]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 04:06:15.469916 4137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e09dc4c75a94810ac73103b0b9fe4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.470639 4136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c188bfc44d644759a415548e24d6bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:15.494623 4344 tablet_bootstrap.cc:492] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:15.536216 4342 tablet_bootstrap.cc:492] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.545517 4342 ts_tablet_manager.cc:1397] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.159s user 0.027s sys 0.027s
I20260806 04:06:15.548529 4342 raft_consensus.cc:348] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.549304 4342 raft_consensus.cc:374] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.549597 4342 raft_consensus.cc:729] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.563024 4342 consensus_queue.cc:260] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.564744 4344 tablet_bootstrap.cc:654] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.574911 4348 tablet_bootstrap.cc:492] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.578809 4320 heartbeater.cc:503] Master 127.2.248.126:33591 was elected leader, sending a full tablet report...
I20260806 04:06:15.579237 4344 tablet_bootstrap.cc:492] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:15.579401 4342 ts_tablet_manager.cc:1428] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.033s user 0.012s sys 0.009s
I20260806 04:06:15.579764 4344 ts_tablet_manager.cc:1397] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.086s user 0.018s sys 0.005s
I20260806 04:06:15.580595 4342 tablet_bootstrap.cc:492] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.581925 4348 tablet_bootstrap.cc:654] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.582669 4344 raft_consensus.cc:348] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.583542 4344 raft_consensus.cc:374] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.583866 4344 raft_consensus.cc:729] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.584692 4344 consensus_queue.cc:260] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.587402 4342 tablet_bootstrap.cc:654] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.594369 4344 ts_tablet_manager.cc:1428] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260806 04:06:15.595183 4348 tablet_bootstrap.cc:492] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.595616 4348 ts_tablet_manager.cc:1397] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20260806 04:06:15.597756 4348 raft_consensus.cc:348] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.598836 4088 heartbeater.cc:503] Master 127.2.248.126:33591 was elected leader, sending a full tablet report...
I20260806 04:06:15.600456 4344 tablet_bootstrap.cc:492] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:15.601583 4342 tablet_bootstrap.cc:492] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.602224 4342 ts_tablet_manager.cc:1397] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.005s
I20260806 04:06:15.598317 4348 raft_consensus.cc:374] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.603569 4348 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.604322 4348 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.608439 4344 tablet_bootstrap.cc:654] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.610206 4349 raft_consensus.cc:482] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.610733 4349 raft_consensus.cc:504] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.613409 4349 leader_election.cc:288] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:15.615134 4342 raft_consensus.cc:348] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.615906 4342 raft_consensus.cc:374] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.616214 4342 raft_consensus.cc:729] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.616911 4342 consensus_queue.cc:260] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.619342 4342 ts_tablet_manager.cc:1428] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260806 04:06:15.620318 4342 tablet_bootstrap.cc:492] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.626403 4342 tablet_bootstrap.cc:654] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.627904 4205 heartbeater.cc:503] Master 127.2.248.126:33591 was elected leader, sending a full tablet report...
I20260806 04:06:15.628424 4348 ts_tablet_manager.cc:1428] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.033s user 0.009s sys 0.018s
I20260806 04:06:15.629455 4348 tablet_bootstrap.cc:492] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.629493 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
W20260806 04:06:15.631978 4229 leader_election.cc:341] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.635931 4348 tablet_bootstrap.cc:654] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.642881 4344 tablet_bootstrap.cc:492] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
W20260806 04:06:15.643101 4206 tablet.cc:2369] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:06:15.643502 4344 ts_tablet_manager.cc:1397] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.004s
I20260806 04:06:15.647110 4344 raft_consensus.cc:348] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.647924 4344 raft_consensus.cc:374] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.648254 4344 raft_consensus.cc:729] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.648892 4348 tablet_bootstrap.cc:492] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.649415 4348 ts_tablet_manager.cc:1397] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20260806 04:06:15.648999 4344 consensus_queue.cc:260] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.651558 4344 ts_tablet_manager.cc:1428] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 04:06:15.652549 4344 tablet_bootstrap.cc:492] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:15.652415 4348 raft_consensus.cc:348] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.653162 4348 raft_consensus.cc:374] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.653587 4348 raft_consensus.cc:729] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.654366 4348 consensus_queue.cc:260] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.656879 4348 ts_tablet_manager.cc:1428] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260806 04:06:15.657821 4348 tablet_bootstrap.cc:492] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.659435 4344 tablet_bootstrap.cc:654] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.668890 4348 tablet_bootstrap.cc:654] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.675410 4342 tablet_bootstrap.cc:492] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.675990 4342 ts_tablet_manager.cc:1397] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.034s
I20260806 04:06:15.676463 4348 tablet_bootstrap.cc:492] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.676946 4348 ts_tablet_manager.cc:1397] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260806 04:06:15.678869 4342 raft_consensus.cc:348] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:15.679697 4342 raft_consensus.cc:374] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.680044 4342 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.680397 4348 raft_consensus.cc:348] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.681458 4348 raft_consensus.cc:374] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.681831 4348 raft_consensus.cc:729] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.680948 4342 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:15.684206 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:15.683962 4348 consensus_queue.cc:260] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.684942 4167 raft_consensus.cc:2454] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 0.
I20260806 04:06:15.684692 4342 ts_tablet_manager.cc:1428] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:06:15.686421 4342 tablet_bootstrap.cc:492] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.686611 4228 leader_election.cc:302] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters: c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:15.687911 4349 raft_consensus.cc:2781] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:15.688277 4349 raft_consensus.cc:482] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:15.688638 4349 raft_consensus.cc:3037] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.691206 4344 tablet_bootstrap.cc:492] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:15.692386 4342 tablet_bootstrap.cc:654] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.692993 4344 ts_tablet_manager.cc:1397] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.019s
I20260806 04:06:15.695721 4344 raft_consensus.cc:348] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.696521 4344 raft_consensus.cc:374] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.696440 4349 raft_consensus.cc:504] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.696949 4344 raft_consensus.cc:729] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.697898 4344 consensus_queue.cc:260] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.686920 4348 ts_tablet_manager.cc:1428] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260806 04:06:15.699677 4348 tablet_bootstrap.cc:492] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.700048 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:15.700464 4344 ts_tablet_manager.cc:1428] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:06:15.701835 4344 tablet_bootstrap.cc:492] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:15.702111 4349 leader_election.cc:288] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:15.702975 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:15.703765 4167 raft_consensus.cc:3037] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.706643 4348 tablet_bootstrap.cc:654] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
W20260806 04:06:15.707177 4229 leader_election.cc:341] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.708379 4344 tablet_bootstrap.cc:654] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.714511 4342 tablet_bootstrap.cc:492] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.715035 4342 ts_tablet_manager.cc:1397] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.008s
I20260806 04:06:15.716305 4348 tablet_bootstrap.cc:492] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.716962 4348 ts_tablet_manager.cc:1397] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20260806 04:06:15.717806 4342 raft_consensus.cc:348] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.718685 4342 raft_consensus.cc:374] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.718992 4342 raft_consensus.cc:729] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.719291 4353 raft_consensus.cc:482] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.719750 4353 raft_consensus.cc:504] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.719811 4342 consensus_queue.cc:260] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.720238 4348 raft_consensus.cc:348] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:15.721103 4348 raft_consensus.cc:374] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.721415 4348 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.722695 4342 ts_tablet_manager.cc:1428] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 04:06:15.722532 4167 raft_consensus.cc:2454] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 1.
I20260806 04:06:15.722334 4348 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:15.723639 4342 tablet_bootstrap.cc:492] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.723886 4228 leader_election.cc:302] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters: c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:15.724735 4348 ts_tablet_manager.cc:1428] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260806 04:06:15.725113 4349 raft_consensus.cc:2781] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:15.725615 4348 tablet_bootstrap.cc:492] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.727291 4353 leader_election.cc:288] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.733512 4348 tablet_bootstrap.cc:654] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.733565 4353 raft_consensus.cc:482] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.734342 4353 raft_consensus.cc:504] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.736475 4353 leader_election.cc:288] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.737910 4342 tablet_bootstrap.cc:654] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.744736 4349 raft_consensus.cc:686] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:15.745947 4349 consensus_queue.cc:237] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.757570 4355 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.758116 4355 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
W20260806 04:06:15.766870 3966 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:15.771852 4355 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.773078 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768c842e242e4b758b52d8f70bd6a3e3" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:15.773729 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
W20260806 04:06:15.774643 3992 leader_election.cc:341] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:15.775424 3992 leader_election.cc:341] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.787252 4348 tablet_bootstrap.cc:492] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.787979 4348 ts_tablet_manager.cc:1397] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.019s
I20260806 04:06:15.789172 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768c842e242e4b758b52d8f70bd6a3e3" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:15.789845 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:15.790638 4282 raft_consensus.cc:2454] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 0.
I20260806 04:06:15.789978 4283 raft_consensus.cc:2454] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 0.
I20260806 04:06:15.793485 4344 tablet_bootstrap.cc:492] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:15.793984 4344 ts_tablet_manager.cc:1397] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.092s user 0.025s sys 0.008s
I20260806 04:06:15.793653 3993 leader_election.cc:302] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:15.793851 4348 raft_consensus.cc:348] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.794828 4348 raft_consensus.cc:374] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.795143 4348 raft_consensus.cc:729] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.795317 4353 raft_consensus.cc:2781] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:15.795810 4353 raft_consensus.cc:482] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:15.796304 4353 raft_consensus.cc:3037] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.795879 4348 consensus_queue.cc:260] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.797020 4344 raft_consensus.cc:348] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.798002 4344 raft_consensus.cc:374] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.798350 4344 raft_consensus.cc:729] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.799271 4344 consensus_queue.cc:260] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.807030 4353 raft_consensus.cc:504] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.811721 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
W20260806 04:06:15.813416 3992 leader_election.cc:341] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.808573 3993 leader_election.cc:302] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:15.815465 4353 leader_election.cc:288] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.820127 3896 catalog_manager.cc:5495] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 0 to 1, leader changed from <none> to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:15.829319 4353 raft_consensus.cc:2781] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:15.830802 4353 raft_consensus.cc:482] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:15.831986 4353 raft_consensus.cc:3037] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.825263 4342 tablet_bootstrap.cc:492] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.834502 4342 ts_tablet_manager.cc:1397] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.111s user 0.024s sys 0.030s
I20260806 04:06:15.837865 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:15.833250 4355 raft_consensus.cc:482] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.833441 4382 raft_consensus.cc:482] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.838987 4355 raft_consensus.cc:504] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.822230 4348 ts_tablet_manager.cc:1428] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260806 04:06:15.840739 4348 tablet_bootstrap.cc:492] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.841203 4355 leader_election.cc:288] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.838225 4342 raft_consensus.cc:348] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.842139 4342 raft_consensus.cc:374] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:06:15.839550 4114 leader_election.cc:341] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.842561 4342 raft_consensus.cc:729] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.843842 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:15.839205 4382 raft_consensus.cc:504] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.846149 4344 ts_tablet_manager.cc:1428] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.052s user 0.014s sys 0.015s
W20260806 04:06:15.846297 4114 leader_election.cc:341] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.846873 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:15.847115 4344 tablet_bootstrap.cc:492] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
W20260806 04:06:15.848456 3992 leader_election.cc:341] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.843696 4342 consensus_queue.cc:260] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.851550 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:15.852697 4382 leader_election.cc:288] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.852828 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:15.852849 4342 ts_tablet_manager.cc:1428] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
W20260806 04:06:15.853793 4114 leader_election.cc:341] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.855330 4342 tablet_bootstrap.cc:492] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.855598 4114 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab; no voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:15.856410 4355 raft_consensus.cc:2726] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:15.856802 4114 leader_election.cc:341] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.857247 4114 leader_election.cc:302] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab; no voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:15.857687 4348 tablet_bootstrap.cc:654] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
W20260806 04:06:15.853248 3993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:41145, user_credentials={real_user=slave}} blocked reactor thread for 44770.8us
I20260806 04:06:15.857992 4355 raft_consensus.cc:2726] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:15.860235 4353 raft_consensus.cc:504] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.860801 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:15.864038 4282 raft_consensus.cc:3037] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.868249 4342 tablet_bootstrap.cc:654] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.869370 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768c842e242e4b758b52d8f70bd6a3e3" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:15.869913 4353 leader_election.cc:288] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.870608 4348 tablet_bootstrap.cc:492] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.862848 4344 tablet_bootstrap.cc:654] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.863138 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
W20260806 04:06:15.870872 3992 leader_election.cc:341] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.871424 4348 ts_tablet_manager.cc:1397] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.007s
I20260806 04:06:15.871462 4283 raft_consensus.cc:2454] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 0.
I20260806 04:06:15.873525 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768c842e242e4b758b52d8f70bd6a3e3" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:15.874105 4283 raft_consensus.cc:3037] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.874509 4348 raft_consensus.cc:348] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.875321 4348 raft_consensus.cc:374] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.875638 4348 raft_consensus.cc:729] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.876600 3993 leader_election.cc:302] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:15.876386 4348 consensus_queue.cc:260] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.878691 4348 ts_tablet_manager.cc:1428] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:06:15.879076 4353 raft_consensus.cc:2781] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:15.879474 4353 raft_consensus.cc:482] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:15.880802 4353 raft_consensus.cc:3037] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.882999 4342 tablet_bootstrap.cc:492] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.883482 4342 ts_tablet_manager.cc:1397] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.002s
I20260806 04:06:15.885883 4342 raft_consensus.cc:348] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.886363 4342 raft_consensus.cc:374] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.886531 4342 raft_consensus.cc:729] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.887276 4342 consensus_queue.cc:260] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.889045 4353 raft_consensus.cc:504] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.889596 4342 ts_tablet_manager.cc:1428] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:06:15.890594 4342 tablet_bootstrap.cc:492] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.891705 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
W20260806 04:06:15.893041 3992 leader_election.cc:341] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.893940 4344 tablet_bootstrap.cc:492] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:15.896574 4342 tablet_bootstrap.cc:654] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.896749 4344 ts_tablet_manager.cc:1397] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.009s
I20260806 04:06:15.898164 4282 raft_consensus.cc:2454] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 1.
I20260806 04:06:15.879724 4348 tablet_bootstrap.cc:492] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.902482 4342 tablet_bootstrap.cc:492] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.903203 4342 ts_tablet_manager.cc:1397] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260806 04:06:15.902776 4344 raft_consensus.cc:348] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.903574 4344 raft_consensus.cc:374] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.903438 3993 leader_election.cc:302] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:15.903939 4344 raft_consensus.cc:729] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.904436 4384 raft_consensus.cc:2781] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:15.904938 4384 raft_consensus.cc:686] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:15.904915 4344 consensus_queue.cc:260] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.905889 4342 raft_consensus.cc:348] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.905937 4384 consensus_queue.cc:237] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.906747 4342 raft_consensus.cc:374] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.907074 4342 raft_consensus.cc:729] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.907347 4344 ts_tablet_manager.cc:1428] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 04:06:15.907873 4342 consensus_queue.cc:260] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.908804 4353 leader_election.cc:288] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:15.908921 4344 tablet_bootstrap.cc:492] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:15.911271 4342 ts_tablet_manager.cc:1428] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:06:15.911652 4348 tablet_bootstrap.cc:654] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.911811 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:15.912266 4342 tablet_bootstrap.cc:492] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.912423 4282 raft_consensus.cc:3037] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.920567 4342 tablet_bootstrap.cc:654] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.902552 4283 raft_consensus.cc:2454] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 1.
I20260806 04:06:15.931723 4282 raft_consensus.cc:2454] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 1.
I20260806 04:06:15.931900 3993 leader_election.cc:302] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:15.933646 3993 leader_election.cc:302] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:15.933084 3898 catalog_manager.cc:5495] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 1 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:15.935092 4382 raft_consensus.cc:2781] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:15.935972 4382 raft_consensus.cc:686] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:15.936859 4382 consensus_queue.cc:237] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.939445 4344 tablet_bootstrap.cc:654] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.939813 4348 tablet_bootstrap.cc:492] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:15.934887 4384 raft_consensus.cc:2781] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:15.940445 4348 ts_tablet_manager.cc:1397] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.061s user 0.024s sys 0.008s
I20260806 04:06:15.941174 4384 raft_consensus.cc:686] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:15.943708 4384 consensus_queue.cc:237] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.946928 4349 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:15.947438 4349 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.949512 4349 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:15.951282 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:15.952008 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:15.952119 4166 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 0.
I20260806 04:06:15.953555 4228 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
W20260806 04:06:15.954392 4229 leader_election.cc:341] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.955315 4349 raft_consensus.cc:2781] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:15.955832 4349 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:15.955322 4348 raft_consensus.cc:348] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.956198 4342 tablet_bootstrap.cc:492] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.956184 4349 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.956457 4348 raft_consensus.cc:374] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.956818 4342 ts_tablet_manager.cc:1397] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.008s
I20260806 04:06:15.956951 4348 raft_consensus.cc:729] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.957898 4348 consensus_queue.cc:260] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.958235 3898 catalog_manager.cc:5495] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 1 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:15.959614 4342 raft_consensus.cc:348] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.960387 4342 raft_consensus.cc:374] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.960656 4342 raft_consensus.cc:729] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.961305 4342 consensus_queue.cc:260] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.963364 4342 ts_tablet_manager.cc:1428] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 04:06:15.964385 4342 tablet_bootstrap.cc:492] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.964767 3898 catalog_manager.cc:5495] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 1 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:15.967787 4349 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.969789 4342 tablet_bootstrap.cc:654] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.970610 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:15.971998 4348 ts_tablet_manager.cc:1428] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.031s user 0.005s sys 0.003s
W20260806 04:06:15.972342 4229 leader_election.cc:341] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:15.973162 4348 tablet_bootstrap.cc:492] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:15.975459 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:15.976064 4166 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:15.979801 4348 tablet_bootstrap.cc:654] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:15.983968 4166 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 1.
I20260806 04:06:15.984279 4342 tablet_bootstrap.cc:492] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:15.984797 4342 ts_tablet_manager.cc:1397] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.005s
I20260806 04:06:15.985162 4228 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters: c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:15.986176 4369 raft_consensus.cc:2781] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:15.987462 4349 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:15.987143 4342 raft_consensus.cc:348] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.988018 4342 raft_consensus.cc:374] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:15.988363 4342 raft_consensus.cc:729] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:15.989331 4342 consensus_queue.cc:260] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.991981 4342 ts_tablet_manager.cc:1428] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 04:06:15.993091 4342 tablet_bootstrap.cc:492] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: Bootstrap starting.
I20260806 04:06:15.997354 4369 raft_consensus.cc:686] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:15.998404 4369 consensus_queue.cc:237] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:15.999789 4344 tablet_bootstrap.cc:492] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:16.000206 4342 tablet_bootstrap.cc:654] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.000260 4344 ts_tablet_manager.cc:1397] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.015s
I20260806 04:06:16.003626 4344 raft_consensus.cc:348] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.004503 4344 raft_consensus.cc:374] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.004864 4344 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.006150 4344 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.008625 4344 ts_tablet_manager.cc:1428] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:06:16.008678 4342 tablet_bootstrap.cc:492] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: No bootstrap required, opened a new log
I20260806 04:06:16.009212 4342 ts_tablet_manager.cc:1397] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260806 04:06:16.009733 4344 tablet_bootstrap.cc:492] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:16.009660 4355 raft_consensus.cc:482] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.010752 4355 raft_consensus.cc:504] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.011648 4342 raft_consensus.cc:348] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.012347 4342 raft_consensus.cc:374] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.012601 4342 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.012753 4355 leader_election.cc:288] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.013413 4342 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.014971 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:16.015430 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.015833 4342 ts_tablet_manager.cc:1428] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:06:16.016171 4282 raft_consensus.cc:2379] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6e3133f3f04620ade89590b792b4ab in current term 1: Already voted for candidate c9e10b44725b4705b9ef52d9b4c856fd in this term.
I20260806 04:06:16.016268 4344 tablet_bootstrap.cc:654] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.017189 4114 leader_election.cc:302] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab; no voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.018824 4355 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.019428 4355 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.020776 4411 raft_consensus.cc:3037] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.022809 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
W20260806 04:06:16.024195 4114 leader_election.cc:341] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:16.024188 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.024919 4282 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 0.
I20260806 04:06:16.025947 4114 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters: c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.021349 4355 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145), c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573)
I20260806 04:06:16.027513 4355 raft_consensus.cc:2781] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:16.027902 4355 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:16.028301 4355 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.031296 4411 raft_consensus.cc:2726] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.036371 4348 tablet_bootstrap.cc:492] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:16.036893 4348 ts_tablet_manager.cc:1397] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.004s
I20260806 04:06:16.040901 3896 catalog_manager.cc:5495] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 0 to 1, leader changed from <none> to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:16.042524 4414 raft_consensus.cc:482] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.042809 4344 tablet_bootstrap.cc:492] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:16.043277 4344 ts_tablet_manager.cc:1397] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.021s
I20260806 04:06:16.043128 4355 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.045188 4355 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 election: Requested vote from peers c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145), c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573)
I20260806 04:06:16.044970 4348 raft_consensus.cc:348] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.043082 4414 raft_consensus.cc:504] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.045643 4348 raft_consensus.cc:374] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.045951 4348 raft_consensus.cc:729] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.045982 4344 raft_consensus.cc:348] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.046804 4344 raft_consensus.cc:374] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.046681 4348 consensus_queue.cc:260] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.047060 4414 leader_election.cc:288] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.047092 4344 raft_consensus.cc:729] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.048801 4348 ts_tablet_manager.cc:1428] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260806 04:06:16.049110 4344 consensus_queue.cc:260] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.050309 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.051152 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.051462 4344 ts_tablet_manager.cc:1428] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 04:06:16.051725 4282 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.052546 4344 tablet_bootstrap.cc:492] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:16.055007 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.055639 4283 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 0.
I20260806 04:06:16.056741 4114 leader_election.cc:302] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:16.057687 4414 raft_consensus.cc:2781] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:16.049595 4348 tablet_bootstrap.cc:492] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:16.058053 4414 raft_consensus.cc:482] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:16.058490 4414 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.056694 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
W20260806 04:06:16.061929 4114 leader_election.cc:341] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:16.062109 4047 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 0.
I20260806 04:06:16.074105 4382 raft_consensus.cc:482] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.074677 4382 raft_consensus.cc:504] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.076630 4382 leader_election.cc:288] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.078231 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
W20260806 04:06:16.080174 3992 leader_election.cc:341] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:16.078258 4344 tablet_bootstrap.cc:654] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.083245 4348 tablet_bootstrap.cc:654] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.086781 4414 raft_consensus.cc:504] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.088665 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.089323 4283 raft_consensus.cc:2454] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 0.
I20260806 04:06:16.090445 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.091225 4047 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.092707 4369 raft_consensus.cc:482] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.092978 4282 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 1.
I20260806 04:06:16.093164 4369 raft_consensus.cc:504] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.094305 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.095515 3993 leader_election.cc:302] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:16.096689 4369 leader_election.cc:288] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.094905 4283 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.098889 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:16.099854 4049 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 0.
I20260806 04:06:16.101104 4229 leader_election.cc:302] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.102437 4369 raft_consensus.cc:2781] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:16.102969 4369 raft_consensus.cc:482] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:16.103361 4382 raft_consensus.cc:2781] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:16.103417 4369 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.103839 4382 raft_consensus.cc:482] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:16.104171 4382 raft_consensus.cc:3037] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.106966 4114 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters: c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.108875 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:16.095472 4414 leader_election.cc:288] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.110105 4355 raft_consensus.cc:2781] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:16.110579 4355 raft_consensus.cc:686] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:16.111397 4355 consensus_queue.cc:237] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.113535 4283 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 1.
I20260806 04:06:16.116336 4114 leader_election.cc:302] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:16.117503 4411 raft_consensus.cc:2781] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader election won for term 1
W20260806 04:06:16.116411 4228 leader_election.cc:341] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:16.120112 4047 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 1.
I20260806 04:06:16.123085 4382 raft_consensus.cc:504] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.127349 4348 tablet_bootstrap.cc:492] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:16.127945 4348 ts_tablet_manager.cc:1397] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.032s
I20260806 04:06:16.128216 4382 leader_election.cc:288] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.128563 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.130065 4344 tablet_bootstrap.cc:492] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:16.134358 4344 ts_tablet_manager.cc:1397] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.082s user 0.025s sys 0.012s
I20260806 04:06:16.133294 4411 raft_consensus.cc:686] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
W20260806 04:06:16.134728 3992 leader_election.cc:341] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:16.135264 4411 consensus_queue.cc:237] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.136404 4348 raft_consensus.cc:348] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.132890 3896 catalog_manager.cc:5495] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 1 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:16.136960 4344 raft_consensus.cc:348] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.137660 4344 raft_consensus.cc:374] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.129639 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.137959 4344 raft_consensus.cc:729] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.133023 4369 raft_consensus.cc:504] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.139672 4283 raft_consensus.cc:3037] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.140225 4344 consensus_queue.cc:260] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.142190 4344 ts_tablet_manager.cc:1428] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:06:16.142210 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.142921 4047 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.143250 4344 tablet_bootstrap.cc:492] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:16.147337 4283 raft_consensus.cc:2454] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 1.
I20260806 04:06:16.137933 4348 raft_consensus.cc:374] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.150959 4348 raft_consensus.cc:729] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.151616 4348 consensus_queue.cc:260] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.152539 3993 leader_election.cc:302] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:16.153703 4348 ts_tablet_manager.cc:1428] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260806 04:06:16.153290 3896 catalog_manager.cc:5495] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 1 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.154525 4348 tablet_bootstrap.cc:492] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab: Bootstrap starting.
I20260806 04:06:16.148864 4344 tablet_bootstrap.cc:654] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.156914 4382 raft_consensus.cc:2781] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:16.157473 4382 raft_consensus.cc:686] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:16.158332 4382 consensus_queue.cc:237] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.163453 4348 tablet_bootstrap.cc:654] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.167244 4047 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 1.
I20260806 04:06:16.168725 4229 leader_election.cc:302] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.169791 4349 raft_consensus.cc:2781] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:16.170392 4349 raft_consensus.cc:686] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.171239 4349 consensus_queue.cc:237] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.173293 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.173879 4166 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.181821 4166 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 1.
I20260806 04:06:16.181357 3898 catalog_manager.cc:5495] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 0 to 1, leader changed from <none> to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:16.183794 4348 tablet_bootstrap.cc:492] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab: No bootstrap required, opened a new log
I20260806 04:06:16.184329 4348 ts_tablet_manager.cc:1397] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.007s
I20260806 04:06:16.183833 4369 leader_election.cc:288] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.188009 4348 raft_consensus.cc:348] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.188706 4348 raft_consensus.cc:374] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.189067 4348 raft_consensus.cc:729] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.190384 4348 consensus_queue.cc:260] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.195214 4348 ts_tablet_manager.cc:1428] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20260806 04:06:16.196352 4344 tablet_bootstrap.cc:492] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:16.196821 4344 ts_tablet_manager.cc:1397] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.017s
I20260806 04:06:16.196493 3896 catalog_manager.cc:5495] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 1 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.198774 4344 raft_consensus.cc:348] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.199419 4344 raft_consensus.cc:374] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.199668 4344 raft_consensus.cc:729] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.200170 4344 consensus_queue.cc:260] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.201872 4344 ts_tablet_manager.cc:1428] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:06:16.202530 4344 tablet_bootstrap.cc:492] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: Bootstrap starting.
I20260806 04:06:16.206599 4344 tablet_bootstrap.cc:654] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:16.211436 4344 tablet_bootstrap.cc:492] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: No bootstrap required, opened a new log
I20260806 04:06:16.211901 4344 ts_tablet_manager.cc:1397] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260806 04:06:16.213765 4344 raft_consensus.cc:348] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.214260 4344 raft_consensus.cc:374] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:16.214473 4344 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Initialized, Role: FOLLOWER
I20260806 04:06:16.214972 4344 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.216285 4344 ts_tablet_manager.cc:1428] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 04:06:16.247612 4382 raft_consensus.cc:482] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.248085 4382 raft_consensus.cc:504] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.249680 4382 leader_election.cc:288] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.250065 4411 raft_consensus.cc:482] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.250829 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.250566 4411 raft_consensus.cc:504] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.250849 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:16.251701 4166 raft_consensus.cc:2379] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9e10b44725b4705b9ef52d9b4c856fd in current term 1: Already voted for candidate c30b4ebfcccd4fd080c384375c5b1528 in this term.
I20260806 04:06:16.252692 4411 leader_election.cc:288] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.252669 3992 leader_election.cc:302] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:16.253392 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768c842e242e4b758b52d8f70bd6a3e3" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:16.253752 4382 raft_consensus.cc:3037] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.253847 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768c842e242e4b758b52d8f70bd6a3e3" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.254673 4283 raft_consensus.cc:2379] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6e3133f3f04620ade89590b792b4ab in current term 1: Already voted for candidate c9e10b44725b4705b9ef52d9b4c856fd in this term.
I20260806 04:06:16.255551 4114 leader_election.cc:302] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab; no voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.256589 4411 raft_consensus.cc:3037] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.260716 4382 raft_consensus.cc:2726] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.262830 4411 raft_consensus.cc:2726] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.272061 4369 raft_consensus.cc:482] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.272516 4369 raft_consensus.cc:504] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.274266 4369 leader_election.cc:288] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.275132 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:16.275454 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:16.275695 4047 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 0.
I20260806 04:06:16.276211 4166 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 0.
I20260806 04:06:16.276722 4229 leader_election.cc:302] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.277535 4369 raft_consensus.cc:2781] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:16.277886 4369 raft_consensus.cc:482] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:16.278154 4369 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.282737 4369 raft_consensus.cc:504] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.284309 4369 leader_election.cc:288] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.285076 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.285331 4166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.285714 4047 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.285881 4166 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.292259 4047 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 1.
I20260806 04:06:16.292259 4166 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 1.
I20260806 04:06:16.293447 4228 leader_election.cc:302] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:16.294062 4369 raft_consensus.cc:2781] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:16.294521 4369 raft_consensus.cc:686] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.295296 4369 consensus_queue.cc:237] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.302026 3897 catalog_manager.cc:5495] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 0 to 1, leader changed from <none> to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:16.327212 4411 raft_consensus.cc:482] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.327685 4411 raft_consensus.cc:504] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.329200 4411 leader_election.cc:288] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.330140 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:16.330351 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.331292 4283 raft_consensus.cc:2379] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6e3133f3f04620ade89590b792b4ab in current term 1: Already voted for candidate c9e10b44725b4705b9ef52d9b4c856fd in this term.
I20260806 04:06:16.332185 4114 leader_election.cc:302] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab; no voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.332978 4411 raft_consensus.cc:3037] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.334558 4349 consensus_queue.cc:1035] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.336985 4382 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.337442 4382 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.339387 4382 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.339732 4411 raft_consensus.cc:2726] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.340505 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.341746 4349 consensus_queue.cc:1035] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.343046 4283 raft_consensus.cc:2379] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9e10b44725b4705b9ef52d9b4c856fd in current term 1: Already voted for candidate 8d6e3133f3f04620ade89590b792b4ab in this term.
I20260806 04:06:16.343329 4384 consensus_queue.cc:1035] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.343910 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:16.347664 3993 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:16.350167 4384 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.360076 4382 consensus_queue.cc:1035] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.375932 4385 consensus_queue.cc:1035] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:16.387578 4385 consensus_queue.cc:1035] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.392467 4382 consensus_queue.cc:1035] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.401491 4384 raft_consensus.cc:2726] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.413689 4382 consensus_queue.cc:1035] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.450976 4382 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.451438 4382 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.453137 4382 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.454131 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6e3133f3f04620ade89590b792b4ab" is_pre_election: true
I20260806 04:06:16.454205 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.454876 4167 raft_consensus.cc:2379] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9e10b44725b4705b9ef52d9b4c856fd in current term 1: Already voted for candidate c30b4ebfcccd4fd080c384375c5b1528 in this term.
I20260806 04:06:16.455758 3992 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9e10b44725b4705b9ef52d9b4c856fd; no voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:16.456434 4382 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.461220 4382 raft_consensus.cc:2726] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.488499 4411 raft_consensus.cc:482] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:16.489032 4411 raft_consensus.cc:504] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.489907 4167 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:59572
I20260806 04:06:16.490511 4167 raft_consensus.cc:595] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.491019 4411 leader_election.cc:288] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.491804 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" is_pre_election: true
I20260806 04:06:16.492004 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528" is_pre_election: true
I20260806 04:06:16.492694 4282 raft_consensus.cc:2379] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6e3133f3f04620ade89590b792b4ab in current term 1: Already voted for candidate c9e10b44725b4705b9ef52d9b4c856fd in this term.
I20260806 04:06:16.493538 4114 leader_election.cc:302] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab; no voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.494172 4411 raft_consensus.cc:3037] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:16.496078 4167 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:59572
I20260806 04:06:16.496632 4167 raft_consensus.cc:595] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.500103 4411 raft_consensus.cc:2726] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:16.515630 4282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46912
I20260806 04:06:16.516188 4282 raft_consensus.cc:595] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:16.521086 4282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
mode: GRACEFUL
new_leader_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:46912
I20260806 04:06:16.521593 4282 raft_consensus.cc:595] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Received request to transfer leadership to TS 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:16.526356 4282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
mode: GRACEFUL
new_leader_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:46912
I20260806 04:06:16.526937 4282 raft_consensus.cc:595] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Received request to transfer leadership to TS 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:16.531636 4282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
mode: GRACEFUL
new_leader_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:46912
I20260806 04:06:16.532217 4282 raft_consensus.cc:595] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Received request to transfer leadership to TS 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:16.546137 4047 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:46222
I20260806 04:06:16.546716 4047 raft_consensus.cc:595] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Received request to transfer leadership to TS 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:16.551460 4047 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:46222
I20260806 04:06:16.552052 4047 raft_consensus.cc:595] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Received request to transfer leadership to TS c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:16.562201 4382 consensus_queue.cc:1035] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.573585 4382 consensus_queue.cc:1035] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.576092 4408 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.578572 4408 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.591363 4353 raft_consensus.cc:979] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:16.591903 4382 raft_consensus.cc:1067] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Signalling peer c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:16.593688 4282 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:46902
I20260806 04:06:16.594312 4282 raft_consensus.cc:482] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.594753 4282 raft_consensus.cc:3037] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.599737 4349 raft_consensus.cc:979] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: : Instructing follower 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:16.600203 4349 raft_consensus.cc:1067] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Signalling peer 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:16.601243 4282 raft_consensus.cc:504] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.602684 4163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:59554
I20260806 04:06:16.603272 4282 leader_election.cc:288] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 2 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.604142 4163 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.604527 4163 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.604630 4165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.605321 4165 raft_consensus.cc:3037] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.606729 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c188bfc44d644759a415548e24d6bf0" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.607429 4047 raft_consensus.cc:3032] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.607801 4047 raft_consensus.cc:729] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:16.608723 4047 consensus_queue.cc:260] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.609939 4047 raft_consensus.cc:3037] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.611912 4163 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.614364 4165 raft_consensus.cc:2454] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 2.
I20260806 04:06:16.615579 4228 leader_election.cc:302] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:16.616521 4369 raft_consensus.cc:2781] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.617023 4369 raft_consensus.cc:686] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.617640 4047 raft_consensus.cc:2454] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 2.
I20260806 04:06:16.618181 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.618042 4369 consensus_queue.cc:237] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.618818 4049 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.617005 4163 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.620247 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.620846 4282 raft_consensus.cc:3032] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.621173 4282 raft_consensus.cc:729] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.621858 4282 consensus_queue.cc:260] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.622948 4282 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.626291 4049 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:16.627216 4114 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.627938 4411 raft_consensus.cc:2781] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.628347 4411 raft_consensus.cc:686] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:16.628703 4282 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:16.629143 4411 consensus_queue.cc:237] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.631605 3898 catalog_manager.cc:5495] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 1 to 2, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:16.636826 4369 consensus_queue.cc:1035] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.636176 3897 catalog_manager.cc:5495] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 1 to 2, leader changed from c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67) to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 2 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.644340 4408 raft_consensus.cc:979] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: : Instructing follower 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:16.644797 4349 raft_consensus.cc:1067] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Signalling peer 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:16.646716 4165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:59554
I20260806 04:06:16.647307 4165 raft_consensus.cc:482] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.647671 4165 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.649883 4408 consensus_queue.cc:1035] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.653755 4420 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:16.656629 4165 raft_consensus.cc:504] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.657760 4163 raft_consensus.cc:1226] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Rejecting Update request from peer c30b4ebfcccd4fd080c384375c5b1528 for earlier term 1. Current term is 2. Ops: []
I20260806 04:06:16.658179 4420 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.664419 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.665031 4047 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.668929 4420 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:16.669492 4165 leader_election.cc:288] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.674130 4408 consensus_queue.cc:1046] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.674950 4349 raft_consensus.cc:3032] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.675274 4349 raft_consensus.cc:729] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.675904 4349 consensus_queue.cc:260] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.676247 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.677173 4349 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.683732 4047 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:16.688194 4114 leader_election.cc:302] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.689328 4420 raft_consensus.cc:2781] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.693336 4420 raft_consensus.cc:686] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:16.694567 4420 consensus_queue.cc:237] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.708384 4282 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:16.709347 3898 catalog_manager.cc:5495] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 1 to 2, leader changed from c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67) to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 2 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.745378 4477 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:16.754841 4477 raft_consensus.cc:979] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:16.755285 4477 raft_consensus.cc:1067] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:16.755980 4420 raft_consensus.cc:979] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:16.756493 4477 raft_consensus.cc:1067] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:16.758626 4047 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46220
I20260806 04:06:16.759238 4047 raft_consensus.cc:482] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.759636 4047 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.762928 4046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46220
I20260806 04:06:16.763537 4046 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.763957 4046 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.771297 4047 raft_consensus.cc:504] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.771831 4046 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.776798 4047 leader_election.cc:288] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 2 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.777032 4165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.777727 4165 raft_consensus.cc:3032] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.778151 4165 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:16.779196 4165 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.780345 4282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.780664 4165 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.780937 4282 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.781571 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.782120 4283 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.788960 4408 raft_consensus.cc:979] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:16.789672 4408 raft_consensus.cc:1067] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:16.789681 4282 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 2.
I20260806 04:06:16.791931 4047 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46212
I20260806 04:06:16.792783 4047 raft_consensus.cc:482] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.793424 4047 raft_consensus.cc:3037] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.793670 4283 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 2.
I20260806 04:06:16.794312 4165 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 2.
I20260806 04:06:16.795941 3993 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.796195 3992 leader_election.cc:302] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.797173 4353 raft_consensus.cc:2781] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.797657 4353 raft_consensus.cc:686] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:16.797705 4382 raft_consensus.cc:2781] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.798317 4382 raft_consensus.cc:686] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:16.798628 4353 consensus_queue.cc:237] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.799189 4382 consensus_queue.cc:237] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.803596 4047 raft_consensus.cc:504] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.805114 4046 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 2 election: Requested vote from peers c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:16.806058 4047 leader_election.cc:288] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 2 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.806973 4349 consensus_queue.cc:1035] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.807520 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.808079 4283 raft_consensus.cc:3032] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.808382 4283 raft_consensus.cc:729] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.809755 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140f74cbd74542aa67b1d5ac7d0588" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.810391 4163 raft_consensus.cc:3037] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.810150 4349 consensus_queue.cc:1035] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:16.810537 4165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:16.809408 4283 consensus_queue.cc:260] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.811405 4165 raft_consensus.cc:3032] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.811949 4165 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:16.812238 4283 raft_consensus.cc:3037] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.812976 4165 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:16.814101 4165 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.823005 4165 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 2.
I20260806 04:06:16.823374 4283 raft_consensus.cc:2454] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 2.
I20260806 04:06:16.824232 3897 catalog_manager.cc:5495] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 1 to 2, leader changed from 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 2 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.827018 3993 leader_election.cc:302] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.829937 4353 raft_consensus.cc:2781] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.828526 3897 catalog_manager.cc:5495] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 1 to 2, leader changed from 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 2 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.830698 4353 raft_consensus.cc:686] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:16.831722 4353 consensus_queue.cc:237] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.838466 4163 raft_consensus.cc:2454] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 2.
I20260806 04:06:16.845268 4408 raft_consensus.cc:979] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: : Instructing follower 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:16.847414 4369 raft_consensus.cc:1067] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Signalling peer 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:16.849799 4163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:59554
I20260806 04:06:16.850452 4163 raft_consensus.cc:482] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:16.850871 4163 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.856521 3897 catalog_manager.cc:5495] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 1 to 2, leader changed from c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 2 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:16.860105 4163 raft_consensus.cc:504] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.861032 4167 raft_consensus.cc:1226] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Rejecting Update request from peer c30b4ebfcccd4fd080c384375c5b1528 for earlier term 1. Current term is 2. Ops: []
I20260806 04:06:16.862110 4163 leader_election.cc:288] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:16.863430 4046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:16.863721 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:16.864048 4046 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.864359 4283 raft_consensus.cc:3032] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:16.864652 4283 raft_consensus.cc:729] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:16.865427 4283 consensus_queue.cc:260] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.866686 4283 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:16.872524 4046 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:16.873880 4114 leader_election.cc:302] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:16.874184 4283 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:16.875005 4476 raft_consensus.cc:2781] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:16.875450 4476 raft_consensus.cc:686] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:16.877592 4476 consensus_queue.cc:237] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:16.885114 3897 catalog_manager.cc:5495] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 1 to 2, leader changed from c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67) to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 2 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:17.036298 4384 raft_consensus.cc:979] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:17.036794 4353 raft_consensus.cc:1067] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Signalling peer 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:17.038517 4167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:59556
I20260806 04:06:17.039072 4167 raft_consensus.cc:482] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:17.039386 4167 raft_consensus.cc:3037] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:17.044194 4167 raft_consensus.cc:504] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:17.045811 4167 leader_election.cc:288] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:17.046969 4046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:17.047483 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e09dc4c75a94810ac73103b0b9fe4dc" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:17.047951 4046 raft_consensus.cc:3032] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:17.048149 4283 raft_consensus.cc:3037] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:17.048302 4046 raft_consensus.cc:729] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:17.049307 4046 consensus_queue.cc:260] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:17.050282 4046 raft_consensus.cc:3037] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:17.053787 4283 raft_consensus.cc:2454] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:17.054992 4114 leader_election.cc:302] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:17.055572 4046 raft_consensus.cc:2454] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:17.055680 4476 raft_consensus.cc:2781] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:17.056254 4476 raft_consensus.cc:686] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:17.057096 4476 consensus_queue.cc:237] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:17.063495 3897 catalog_manager.cc:5495] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 1 to 2, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 2 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:17.069531 4046 raft_consensus.cc:1261] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.070905 4369 consensus_queue.cc:1035] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.077811 4167 raft_consensus.cc:1261] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.079447 4408 consensus_queue.cc:1035] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.107844 4046 raft_consensus.cc:1261] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.109117 4476 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.116066 4283 raft_consensus.cc:1261] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.118407 4476 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.133293 4046 raft_consensus.cc:1261] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.134451 4476 consensus_queue.cc:1035] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.142555 4283 raft_consensus.cc:1261] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.144443 4477 consensus_queue.cc:1035] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.193478 4283 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.194725 4353 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.202497 4167 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.204327 4353 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.292884 4167 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.294243 4353 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.297523 4046 raft_consensus.cc:1261] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.298938 4477 consensus_queue.cc:1035] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.306241 4283 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.308290 4385 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:06:17.317955 4282 raft_consensus.cc:1261] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.320461 4477 consensus_queue.cc:1035] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.384033 4167 raft_consensus.cc:1261] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.385425 4385 consensus_queue.cc:1035] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.393177 4282 raft_consensus.cc:1261] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.396867 4385 consensus_queue.cc:1035] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:06:17.534677 4046 raft_consensus.cc:1261] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.536069 4477 consensus_queue.cc:1035] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:17.545394 4282 raft_consensus.cc:1261] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:17.549045 4477 consensus_queue.cc:1035] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:19.585268 3041 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260806 04:06:19.586166 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:19.604911 4046 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:38424
I20260806 04:06:19.605463 4046 raft_consensus.cc:595] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Received request to transfer leadership to TS c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:19.607883 4046 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:38424
I20260806 04:06:19.608428 4046 raft_consensus.cc:595] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Received request to transfer leadership to TS c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:19.609442 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:06:19.609932 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:19.920657 4385 raft_consensus.cc:979] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:19.921094 4493 raft_consensus.cc:1067] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Signalling peer c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:19.922770 4283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:46902
I20260806 04:06:19.923233 4283 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:19.923471 4283 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:19.928989 4283 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:19.930615 4283 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 3 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:19.931748 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:19.931782 4046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:19.932454 4167 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:19.932539 4046 raft_consensus.cc:3032] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:19.932893 4046 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:19.933532 4046 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:19.934458 4046 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:19.937381 4167 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 3.
I20260806 04:06:19.938499 4228 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:19.939293 4484 raft_consensus.cc:2781] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:19.939463 4046 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 3.
I20260806 04:06:19.940709 4484 raft_consensus.cc:686] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:19.941464 4484 consensus_queue.cc:237] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:19.949013 3897 catalog_manager.cc:5495] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 2 to 3, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 3 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:19.991657 4385 raft_consensus.cc:979] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:19.992134 4493 raft_consensus.cc:1067] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Signalling peer c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:19.993680 4283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:46902
I20260806 04:06:19.994225 4283 raft_consensus.cc:482] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:19.994510 4283 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:19.999248 4283 raft_consensus.cc:504] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:20.000911 4283 leader_election.cc:288] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 3 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:20.001941 4167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:20.001947 4046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:20.002538 4167 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:20.002638 4046 raft_consensus.cc:3032] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:20.003015 4046 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:20.003706 4046 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:20.004666 4046 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:20.007575 4167 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 3.
I20260806 04:06:20.008601 4228 leader_election.cc:302] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:20.009308 4484 raft_consensus.cc:2781] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:20.009819 4484 raft_consensus.cc:686] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:20.010004 4046 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 3.
I20260806 04:06:20.010918 4484 consensus_queue.cc:237] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:20.018139 3897 catalog_manager.cc:5495] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 2 to 3, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 3 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:20.370745 4167 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:20.372159 4484 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:20.395394 4046 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:20.397037 4509 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:20.530841 4167 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:20.532195 4509 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:20.536706 4046 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:20.538115 4509 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:20.611052 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:20.616986 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:20.617403 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:21.618229 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:21.623497 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:21.623916 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:22.625069 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:22.630757 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:22.631268 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:23.632145 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:23.638870 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:23.639452 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:24.640295 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:24.646624 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:24.647083 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:25.647801 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:25.653067 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:25.653496 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:26.654320 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:26.659719 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:26.660156 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:27.661044 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:27.666976 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:27.667418 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:28.668154 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:28.673465 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:28.673848 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:29.702188 3041 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260806 04:06:29.706465 4283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46912
I20260806 04:06:29.707135 4283 raft_consensus.cc:595] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:29.711553 4283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46912
I20260806 04:06:29.711992 4283 raft_consensus.cc:595] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:29.722780 4163 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:59572
I20260806 04:06:29.723270 4163 raft_consensus.cc:595] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:29.727763 4163 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:59572
I20260806 04:06:29.728304 4163 raft_consensus.cc:595] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:29.732813 4163 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
mode: GRACEFUL
new_leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:59572
I20260806 04:06:29.733258 4163 raft_consensus.cc:595] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Received request to transfer leadership to TS c9e10b44725b4705b9ef52d9b4c856fd
I20260806 04:06:29.799988 4533 raft_consensus.cc:979] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:29.800561 4534 raft_consensus.cc:1067] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:29.802089 4046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46220
I20260806 04:06:29.802626 4046 raft_consensus.cc:482] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:29.803038 4046 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:29.809199 4046 raft_consensus.cc:504] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:29.811290 4046 leader_election.cc:288] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 3 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:29.812165 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:29.812496 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cc94dc4ae64fdbb3a48e0ff9d4b519" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:29.812974 4163 raft_consensus.cc:3032] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:29.813054 4283 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:29.813266 4163 raft_consensus.cc:729] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:29.814214 4163 consensus_queue.cc:260] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:29.815471 4163 raft_consensus.cc:3037] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:29.819033 4283 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 3.
I20260806 04:06:29.820192 3993 leader_election.cc:302] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:29.821094 4524 raft_consensus.cc:2781] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:29.821846 4163 raft_consensus.cc:2454] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 3.
I20260806 04:06:29.822240 4524 raft_consensus.cc:686] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:29.823603 4524 consensus_queue.cc:237] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:29.830828 3898 catalog_manager.cc:5495] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 2 to 3, leader changed from 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 3 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:30.016933 4529 raft_consensus.cc:979] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.017472 4544 raft_consensus.cc:1067] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.018970 4046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46212
I20260806 04:06:30.019501 4046 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:30.019796 4046 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:30.025386 4046 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:30.026939 4046 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 4 election: Requested vote from peers c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:30.027801 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:30.027910 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:30.028467 4283 raft_consensus.cc:3032] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:30.028506 4163 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:30.028930 4283 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:30.029688 4283 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:30.030977 4283 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:30.035929 4163 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 4.
I20260806 04:06:30.037344 3992 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:30.038113 4524 raft_consensus.cc:2781] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:30.038491 4283 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 4.
I20260806 04:06:30.038640 4524 raft_consensus.cc:686] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:30.039592 4524 consensus_queue.cc:237] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:30.047362 3898 catalog_manager.cc:5495] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 3 to 4, leader changed from c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 4 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:30.157125 4533 raft_consensus.cc:979] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.157579 4534 raft_consensus.cc:1067] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.158919 4046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46220
I20260806 04:06:30.159355 4046 raft_consensus.cc:482] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:30.159602 4046 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:30.165073 4046 raft_consensus.cc:504] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.166743 4046 leader_election.cc:288] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 3 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:30.167398 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:30.167711 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7265c5ca1448adafab545d07ffef39" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:30.168007 4163 raft_consensus.cc:3032] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:30.168289 4283 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:30.168363 4163 raft_consensus.cc:729] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:30.169242 4163 consensus_queue.cc:260] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.170169 4163 raft_consensus.cc:3037] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:30.172912 4283 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 3.
I20260806 04:06:30.173835 3993 leader_election.cc:302] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:30.174834 4536 raft_consensus.cc:2781] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:30.175246 4536 raft_consensus.cc:686] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:30.175310 4163 raft_consensus.cc:2454] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 3.
I20260806 04:06:30.176162 4536 consensus_queue.cc:237] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.183100 3898 catalog_manager.cc:5495] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 2 to 3, leader changed from 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 3 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:30.232313 4163 raft_consensus.cc:1261] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:30.233605 4536 consensus_queue.cc:1035] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:30.244727 4283 raft_consensus.cc:1261] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:30.247921 4550 consensus_queue.cc:1035] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:30.257606 4544 raft_consensus.cc:979] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.262089 4544 raft_consensus.cc:1067] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.265861 4047 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46212
I20260806 04:06:30.266434 4047 raft_consensus.cc:482] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:30.266795 4047 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:30.273164 4047 raft_consensus.cc:504] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.274945 4047 leader_election.cc:288] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 4 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:30.275811 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:30.276203 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:30.276432 4163 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:30.276754 4283 raft_consensus.cc:3032] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:30.277134 4283 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:30.277866 4283 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.279008 4283 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:30.282554 4163 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 4.
I20260806 04:06:30.283353 4283 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 4.
I20260806 04:06:30.283808 3992 leader_election.cc:302] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:30.284581 4524 raft_consensus.cc:2781] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:30.285030 4524 raft_consensus.cc:686] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:30.285833 4524 consensus_queue.cc:237] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.293932 3898 catalog_manager.cc:5495] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 3 to 4, leader changed from c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 4 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:30.301173 4533 raft_consensus.cc:979] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: : Instructing follower c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.301710 4533 raft_consensus.cc:1067] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Signalling peer c9e10b44725b4705b9ef52d9b4c856fd to start an election
I20260806 04:06:30.303435 4047 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
from {username='slave'} at 127.0.0.1:46220
I20260806 04:06:30.303831 4047 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:30.304056 4047 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:30.308586 4047 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.309937 4047 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 3 election: Requested vote from peers 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:30.311154 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:30.311159 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:30.311810 4283 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:30.311888 4163 raft_consensus.cc:3032] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:06:30.312285 4163 raft_consensus.cc:729] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:30.313117 4163 consensus_queue.cc:260] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.314380 4163 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:30.316457 4283 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 3.
I20260806 04:06:30.317446 3993 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30b4ebfcccd4fd080c384375c5b1528, c9e10b44725b4705b9ef52d9b4c856fd; no voters:
I20260806 04:06:30.318337 4524 raft_consensus.cc:2781] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:30.318989 4524 raft_consensus.cc:686] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Becoming Leader. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:30.320009 4524 consensus_queue.cc:237] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:30.321000 4163 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e10b44725b4705b9ef52d9b4c856fd in term 3.
I20260806 04:06:30.328416 3898 catalog_manager.cc:5495] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd reported cstate change: term changed from 2 to 3, leader changed from 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66) to c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65). New cstate: current_term: 3 leader_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:30.486631 4283 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:30.488061 4524 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:30.501446 4163 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:30.502918 4536 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:30.583093 4163 raft_consensus.cc:1261] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:30.584450 4536 consensus_queue.cc:1035] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:30.595517 4283 raft_consensus.cc:1261] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:30.596859 4524 consensus_queue.cc:1035] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:30.720221 4283 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:30.721578 4524 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:30.732844 4163 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:30.734086 4536 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:30.747119 4283 raft_consensus.cc:1261] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:30.748399 4536 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:06:30.760011 4163 raft_consensus.cc:1261] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Refusing update from remote peer c9e10b44725b4705b9ef52d9b4c856fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:06:30.761511 4536 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:32.770372 3041 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260806 04:06:32.771260 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:32.778625 4047 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:38424
I20260806 04:06:32.779812 4047 raft_consensus.cc:595] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Received request to transfer leadership to TS c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:32.782517 4047 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:38424
I20260806 04:06:32.783041 4047 raft_consensus.cc:595] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Received request to transfer leadership to TS 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:32.785457 4047 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:38424
I20260806 04:06:32.785943 4047 raft_consensus.cc:595] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Received request to transfer leadership to TS 8d6e3133f3f04620ade89590b792b4ab
I20260806 04:06:32.788416 4047 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
mode: GRACEFUL
new_leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:38424
I20260806 04:06:32.788935 4047 raft_consensus.cc:595] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Received request to transfer leadership to TS c30b4ebfcccd4fd080c384375c5b1528
I20260806 04:06:32.789898 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:06:32.790562 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:32.791764 4571 raft_consensus.cc:979] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:32.792289 4550 raft_consensus.cc:1067] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Signalling peer 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:32.793658 4163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:59556
I20260806 04:06:32.794082 4163 raft_consensus.cc:482] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:32.794407 4163 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:32.800160 4163 raft_consensus.cc:504] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:32.802161 4163 leader_election.cc:288] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 4 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:32.803212 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:32.803521 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb6388c650c430c89eee0ee05eae6d1" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:32.803853 4047 raft_consensus.cc:3032] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:06:32.804049 4283 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:32.804191 4047 raft_consensus.cc:729] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:32.805050 4047 consensus_queue.cc:260] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:32.806051 4047 raft_consensus.cc:3037] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:06:32.809479 4283 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 4.
I20260806 04:06:32.810796 4114 leader_election.cc:302] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:32.811568 4047 raft_consensus.cc:2454] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 4.
I20260806 04:06:32.812438 4572 raft_consensus.cc:2781] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:06:32.813303 4572 raft_consensus.cc:686] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 4 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:32.814091 4572 consensus_queue.cc:237] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:32.820843 3897 catalog_manager.cc:5495] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 3 to 4, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 4 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:32.830960 4550 raft_consensus.cc:979] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:32.831524 4571 raft_consensus.cc:1067] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Signalling peer c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:32.833021 4283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:46902
I20260806 04:06:32.833503 4283 raft_consensus.cc:482] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:32.833809 4283 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:32.840108 4283 raft_consensus.cc:504] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:32.842027 4283 leader_election.cc:288] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 5 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:32.842799 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:32.843395 4047 raft_consensus.cc:3032] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:06:32.843294 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e64f9fe31af460aab651551bcce4915" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:32.843693 4047 raft_consensus.cc:729] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:32.843839 4163 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:32.844455 4047 consensus_queue.cc:260] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:32.845695 4047 raft_consensus.cc:3037] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:32.850266 4163 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 5.
I20260806 04:06:32.851506 4228 leader_election.cc:302] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:32.852077 4047 raft_consensus.cc:2454] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 5.
I20260806 04:06:32.852298 4570 raft_consensus.cc:2781] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:06:32.853870 4570 raft_consensus.cc:686] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 5 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:32.854782 4570 consensus_queue.cc:237] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } }
I20260806 04:06:32.862216 3897 catalog_manager.cc:5495] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 4 to 5, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 5 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:33.028326 4550 raft_consensus.cc:979] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:33.028786 4571 raft_consensus.cc:1067] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Signalling peer c30b4ebfcccd4fd080c384375c5b1528 to start an election
I20260806 04:06:33.030174 4283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd"
dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
from {username='slave'} at 127.0.0.1:46902
I20260806 04:06:33.030596 4283 raft_consensus.cc:482] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:33.030975 4283 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:33.036461 4283 raft_consensus.cc:504] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:33.037771 4283 leader_election.cc:288] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 5 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015)
I20260806 04:06:33.038626 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:33.038825 4163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d766a61fb5e47059f39a048dd05cbfd" candidate_uuid: "c30b4ebfcccd4fd080c384375c5b1528" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
I20260806 04:06:33.039145 4047 raft_consensus.cc:3032] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:06:33.039363 4163 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:33.039441 4047 raft_consensus.cc:729] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:33.040345 4047 consensus_queue.cc:260] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:33.041635 4047 raft_consensus.cc:3037] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:06:33.045004 4163 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 5.
I20260806 04:06:33.045991 4228 leader_election.cc:302] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:33.046581 4047 raft_consensus.cc:2454] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b4ebfcccd4fd080c384375c5b1528 in term 5.
I20260806 04:06:33.046839 4570 raft_consensus.cc:2781] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:06:33.047372 4570 raft_consensus.cc:686] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 5 LEADER]: Becoming Leader. State: Replica: c30b4ebfcccd4fd080c384375c5b1528, State: Running, Role: LEADER
I20260806 04:06:33.048142 4570 consensus_queue.cc:237] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:33.056185 4550 raft_consensus.cc:979] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: : Instructing follower 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:33.056797 4571 raft_consensus.cc:1067] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Signalling peer 8d6e3133f3f04620ade89590b792b4ab to start an election
I20260806 04:06:33.056214 3897 catalog_manager.cc:5495] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 reported cstate change: term changed from 4 to 5, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67). New cstate: current_term: 5 leader_uuid: "c30b4ebfcccd4fd080c384375c5b1528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:33.058199 4163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731"
dest_uuid: "8d6e3133f3f04620ade89590b792b4ab"
from {username='slave'} at 127.0.0.1:59556
I20260806 04:06:33.058736 4163 raft_consensus.cc:482] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:06:33.059105 4163 raft_consensus.cc:3037] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:33.063817 4163 raft_consensus.cc:504] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:33.065533 4163 leader_election.cc:288] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Requested vote from peers c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573), c30b4ebfcccd4fd080c384375c5b1528 (127.2.248.67:41145)
I20260806 04:06:33.066341 4047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9e10b44725b4705b9ef52d9b4c856fd"
I20260806 04:06:33.066545 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf6122c7b2a443b9c8026e303cab731" candidate_uuid: "8d6e3133f3f04620ade89590b792b4ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c30b4ebfcccd4fd080c384375c5b1528"
I20260806 04:06:33.066962 4047 raft_consensus.cc:3032] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:33.067042 4283 raft_consensus.cc:3037] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:33.067303 4047 raft_consensus.cc:729] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9e10b44725b4705b9ef52d9b4c856fd, State: Running, Role: LEADER
I20260806 04:06:33.068010 4047 consensus_queue.cc:260] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:33.069195 4047 raft_consensus.cc:3037] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:33.073736 4283 raft_consensus.cc:2454] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:33.074878 4114 leader_election.cc:302] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6e3133f3f04620ade89590b792b4ab, c30b4ebfcccd4fd080c384375c5b1528; no voters:
I20260806 04:06:33.075639 4572 raft_consensus.cc:2781] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:06:33.075794 4047 raft_consensus.cc:2454] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6e3133f3f04620ade89590b792b4ab in term 2.
I20260806 04:06:33.076095 4572 raft_consensus.cc:686] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Becoming Leader. State: Replica: 8d6e3133f3f04620ade89590b792b4ab, State: Running, Role: LEADER
I20260806 04:06:33.077039 4572 consensus_queue.cc:237] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } }
I20260806 04:06:33.084398 3897 catalog_manager.cc:5495] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab reported cstate change: term changed from 1 to 2, leader changed from c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65) to 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66). New cstate: current_term: 2 leader_uuid: "8d6e3133f3f04620ade89590b792b4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:33.378051 4047 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:06:33.379321 4570 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:06:33.390559 4163 raft_consensus.cc:1261] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:06:33.396663 4596 consensus_queue.cc:1035] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:06:33.429509 4047 raft_consensus.cc:1261] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:33.430167 4283 raft_consensus.cc:1261] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:06:33.430965 4575 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:33.431945 4572 consensus_queue.cc:1035] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:06:33.534403 4047 raft_consensus.cc:1261] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:33.535655 4604 consensus_queue.cc:1035] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:33.544443 4283 raft_consensus.cc:1261] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Refusing update from remote peer 8d6e3133f3f04620ade89590b792b4ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:06:33.546932 4604 consensus_queue.cc:1035] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b4ebfcccd4fd080c384375c5b1528" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:06:33.565498 4163 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:06:33.566723 4596 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6e3133f3f04620ade89590b792b4ab" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:06:33.574967 4047 raft_consensus.cc:1261] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Refusing update from remote peer c30b4ebfcccd4fd080c384375c5b1528: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:06:33.577088 4570 consensus_queue.cc:1035] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e10b44725b4705b9ef52d9b4c856fd" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:06:33.791540 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:33.796703 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:33.797174 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:34.798137 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:34.803675 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:34.804150 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:35.805059 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:35.810309 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:35.810858 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:36.811846 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:36.817268 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:36.817657 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:37.818547 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:37.823830 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:37.824235 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:38.825146 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:38.830330 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:38.830869 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:39.831781 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:39.837174 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:39.837702 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:40.838584 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:40.844089 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:40.844528 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:41.846194 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:06:41.854279 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:06:41.854807 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:06:42.885133 3041 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260806 04:06:42.885703 3041 tablet_server.cc:178] TabletServer@127.2.248.65:0 shutting down...
I20260806 04:06:42.917596 3041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:06:42.918411 3041 tablet_replica.cc:330] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.919245 3041 raft_consensus.cc:2229] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:42.920025 3041 raft_consensus.cc:2258] T 5e64f9fe31af460aab651551bcce4915 P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.922174 3041 tablet_replica.cc:330] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.922736 3041 raft_consensus.cc:2229] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Raft consensus shutting down.
I20260806 04:06:42.923631 3041 raft_consensus.cc:2258] T 6c7265c5ca1448adafab545d07ffef39 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.925544 3041 tablet_replica.cc:330] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.926041 3041 raft_consensus.cc:2229] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:06:42.926949 3041 raft_consensus.cc:2258] T 24140f74cbd74542aa67b1d5ac7d0588 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.928711 3041 tablet_replica.cc:330] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.929215 3041 raft_consensus.cc:2229] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:42.929656 3041 raft_consensus.cc:2258] T 8bb6388c650c430c89eee0ee05eae6d1 P c9e10b44725b4705b9ef52d9b4c856fd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.931648 3041 tablet_replica.cc:330] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.932157 3041 raft_consensus.cc:2229] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:42.932588 3041 raft_consensus.cc:2258] T 8d766a61fb5e47059f39a048dd05cbfd P c9e10b44725b4705b9ef52d9b4c856fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.934369 3041 tablet_replica.cc:330] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.934965 3041 raft_consensus.cc:2229] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:42.935464 3041 raft_consensus.cc:2258] T 5c188bfc44d644759a415548e24d6bf0 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.937222 3041 tablet_replica.cc:330] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.937709 3041 raft_consensus.cc:2229] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 LEADER]: Raft consensus shutting down.
I20260806 04:06:42.938593 3041 raft_consensus.cc:2258] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c9e10b44725b4705b9ef52d9b4c856fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.940472 3041 tablet_replica.cc:330] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.940956 3041 raft_consensus.cc:2229] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:06:42.941789 3041 raft_consensus.cc:2258] T 768c842e242e4b758b52d8f70bd6a3e3 P c9e10b44725b4705b9ef52d9b4c856fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.944216 3041 tablet_replica.cc:330] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.944703 3041 raft_consensus.cc:2229] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus shutting down.
W20260806 04:06:42.944751 4114 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
I20260806 04:06:42.945252 3041 raft_consensus.cc:2258] T 7cf6122c7b2a443b9c8026e303cab731 P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:42.947526 3041 tablet_replica.cc:330] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd: stopping tablet replica
I20260806 04:06:42.947993 3041 raft_consensus.cc:2229] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus shutting down.
W20260806 04:06:42.948417 4114 consensus_peers.cc:487] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab -> Peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Couldn't send request to peer c9e10b44725b4705b9ef52d9b4c856fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:42.948515 3041 raft_consensus.cc:2258] T 9e09dc4c75a94810ac73103b0b9fe4dc P c9e10b44725b4705b9ef52d9b4c856fd [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 04:06:42.975168 4229 consensus_peers.cc:487] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 -> Peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Couldn't send request to peer c9e10b44725b4705b9ef52d9b4c856fd. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.65:46573: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:42.980576 3041 tablet_server.cc:194] TabletServer@127.2.248.65:0 shutdown complete.
I20260806 04:06:43.001209 3041 tablet_server.cc:178] TabletServer@127.2.248.66:0 shutting down...
W20260806 04:06:43.028452 4228 consensus_peers.cc:487] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 -> Peer 8d6e3133f3f04620ade89590b792b4ab (127.2.248.66:46015): Couldn't send request to peer 8d6e3133f3f04620ade89590b792b4ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:43.033406 3041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:06:43.034237 3041 tablet_replica.cc:330] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.035178 3041 raft_consensus.cc:2229] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.035815 3041 raft_consensus.cc:2258] T 5c188bfc44d644759a415548e24d6bf0 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.038152 3041 tablet_replica.cc:330] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.038887 3041 raft_consensus.cc:2229] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.039438 3041 raft_consensus.cc:2258] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Raft consensus is shut down!
W20260806 04:06:43.041170 4114 consensus_peers.cc:487] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab -> Peer c9e10b44725b4705b9ef52d9b4c856fd (127.2.248.65:46573): Couldn't send request to peer c9e10b44725b4705b9ef52d9b4c856fd. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.65:46573: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:43.041566 3041 tablet_replica.cc:330] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.042127 3041 raft_consensus.cc:2229] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.042546 3041 raft_consensus.cc:2258] T 768c842e242e4b758b52d8f70bd6a3e3 P 8d6e3133f3f04620ade89590b792b4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.044368 3041 tablet_replica.cc:330] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.044862 3041 raft_consensus.cc:2229] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.045650 3041 raft_consensus.cc:2258] T 7cf6122c7b2a443b9c8026e303cab731 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.047401 3041 tablet_replica.cc:330] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.047909 3041 raft_consensus.cc:2229] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.048671 3041 raft_consensus.cc:2258] T 9e09dc4c75a94810ac73103b0b9fe4dc P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.050330 3041 tablet_replica.cc:330] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.050897 3041 raft_consensus.cc:2229] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.051393 3041 raft_consensus.cc:2258] T 5e64f9fe31af460aab651551bcce4915 P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.053444 3041 tablet_replica.cc:330] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.053973 3041 raft_consensus.cc:2229] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.054472 3041 raft_consensus.cc:2258] T 8d766a61fb5e47059f39a048dd05cbfd P 8d6e3133f3f04620ade89590b792b4ab [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.056191 3041 tablet_replica.cc:330] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.056674 3041 raft_consensus.cc:2229] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.057546 3041 raft_consensus.cc:2258] T 8bb6388c650c430c89eee0ee05eae6d1 P 8d6e3133f3f04620ade89590b792b4ab [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.059288 3041 tablet_replica.cc:330] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.059736 3041 raft_consensus.cc:2229] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.060180 3041 raft_consensus.cc:2258] T 24140f74cbd74542aa67b1d5ac7d0588 P 8d6e3133f3f04620ade89590b792b4ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.061704 3041 tablet_replica.cc:330] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab: stopping tablet replica
I20260806 04:06:43.062134 3041 raft_consensus.cc:2229] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.062577 3041 raft_consensus.cc:2258] T 6c7265c5ca1448adafab545d07ffef39 P 8d6e3133f3f04620ade89590b792b4ab [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.093544 3041 tablet_server.cc:194] TabletServer@127.2.248.66:0 shutdown complete.
I20260806 04:06:43.114113 3041 tablet_server.cc:178] TabletServer@127.2.248.67:0 shutting down...
I20260806 04:06:43.143283 3041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:06:43.144071 3041 tablet_replica.cc:330] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.144696 3041 raft_consensus.cc:2229] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.145560 3041 raft_consensus.cc:2258] T 5c188bfc44d644759a415548e24d6bf0 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.147433 3041 tablet_replica.cc:330] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.147953 3041 raft_consensus.cc:2229] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.148445 3041 raft_consensus.cc:2258] T 81cc94dc4ae64fdbb3a48e0ff9d4b519 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.150035 3041 tablet_replica.cc:330] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.150506 3041 raft_consensus.cc:2229] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.151037 3041 raft_consensus.cc:2258] T 6c7265c5ca1448adafab545d07ffef39 P c30b4ebfcccd4fd080c384375c5b1528 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.152750 3041 tablet_replica.cc:330] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.153256 3041 raft_consensus.cc:2229] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.153723 3041 raft_consensus.cc:2258] T 768c842e242e4b758b52d8f70bd6a3e3 P c30b4ebfcccd4fd080c384375c5b1528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.155506 3041 tablet_replica.cc:330] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.155977 3041 raft_consensus.cc:2229] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 5 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.156767 3041 raft_consensus.cc:2258] T 5e64f9fe31af460aab651551bcce4915 P c30b4ebfcccd4fd080c384375c5b1528 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.158442 3041 tablet_replica.cc:330] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.158938 3041 raft_consensus.cc:2229] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.159404 3041 raft_consensus.cc:2258] T 7cf6122c7b2a443b9c8026e303cab731 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.161155 3041 tablet_replica.cc:330] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.161610 3041 raft_consensus.cc:2229] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.162009 3041 raft_consensus.cc:2258] T 9e09dc4c75a94810ac73103b0b9fe4dc P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.163650 3041 tablet_replica.cc:330] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.164137 3041 raft_consensus.cc:2229] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 5 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.165138 3041 raft_consensus.cc:2258] T 8d766a61fb5e47059f39a048dd05cbfd P c30b4ebfcccd4fd080c384375c5b1528 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.166998 3041 tablet_replica.cc:330] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.167500 3041 raft_consensus.cc:2229] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.167933 3041 raft_consensus.cc:2258] T 8bb6388c650c430c89eee0ee05eae6d1 P c30b4ebfcccd4fd080c384375c5b1528 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.169463 3041 tablet_replica.cc:330] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528: stopping tablet replica
I20260806 04:06:43.169889 3041 raft_consensus.cc:2229] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:43.170270 3041 raft_consensus.cc:2258] T 24140f74cbd74542aa67b1d5ac7d0588 P c30b4ebfcccd4fd080c384375c5b1528 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.199997 3041 tablet_server.cc:194] TabletServer@127.2.248.67:0 shutdown complete.
I20260806 04:06:43.220007 3041 master.cc:537] Master@127.2.248.126:33591 shutting down...
I20260806 04:06:43.245247 3041 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:06:43.245985 3041 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:43.246392 3041 tablet_replica.cc:330] T 00000000000000000000000000000000 P c2b6d374c8d8472c8fd3bb0253ca3508: stopping tablet replica
I20260806 04:06:43.265657 3041 master.cc:552] Master@127.2.248.126:33591 shutdown complete.
8d6e3133f3f04620ade89590b792b4ab 4
c30b4ebfcccd4fd080c384375c5b1528 1
c9e10b44725b4705b9ef52d9b4c856fd 5
8d6e3133f3f04620ade89590b792b4ab 4
c30b4ebfcccd4fd080c384375c5b1528 3
c9e10b44725b4705b9ef52d9b4c856fd 3
8d6e3133f3f04620ade89590b792b4ab 1
c30b4ebfcccd4fd080c384375c5b1528 1
c9e10b44725b4705b9ef52d9b4c856fd 8
8d6e3133f3f04620ade89590b792b4ab 3
c30b4ebfcccd4fd080c384375c5b1528 3
c9e10b44725b4705b9ef52d9b4c856fd 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28680 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260806 04:06:43.309156 3041 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.2.248.126:44289
I20260806 04:06:43.310276 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:43.316179 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:43.316313 4639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:43.317756 4642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:43.318859 3041 server_base.cc:1034] running on GCE node
I20260806 04:06:43.320096 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:43.320284 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:43.320394 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989203320385 us; error 0 us; skew 500 ppm
I20260806 04:06:43.321113 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:43.328604 3041 webserver.cc:460] Webserver started at http://127.2.248.126:42545/ using document root <none> and password file <none>
I20260806 04:06:43.329082 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:43.329257 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:43.329502 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:43.330915 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/master-0-root/instance:
uuid: "7a82866733fa43d09abd13eedc5ee6df"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.335732 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 04:06:43.339149 4650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.339949 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260806 04:06:43.340250 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/master-0-root
uuid: "7a82866733fa43d09abd13eedc5ee6df"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.340549 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:43.364877 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:43.366358 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:43.406090 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.126:44289
I20260806 04:06:43.406176 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.126:44289 every 8 connection(s)
I20260806 04:06:43.410058 4725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:43.420352 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Bootstrap starting.
I20260806 04:06:43.424752 4725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:43.429116 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: No bootstrap required, opened a new log
I20260806 04:06:43.431253 4725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a82866733fa43d09abd13eedc5ee6df" member_type: VOTER }
I20260806 04:06:43.431777 4725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:43.432018 4725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a82866733fa43d09abd13eedc5ee6df, State: Initialized, Role: FOLLOWER
I20260806 04:06:43.432607 4725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a82866733fa43d09abd13eedc5ee6df" member_type: VOTER }
I20260806 04:06:43.433048 4725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:06:43.433324 4725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:06:43.433631 4725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:43.437570 4725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a82866733fa43d09abd13eedc5ee6df" member_type: VOTER }
I20260806 04:06:43.438090 4725 leader_election.cc:302] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a82866733fa43d09abd13eedc5ee6df; no voters:
I20260806 04:06:43.439267 4725 leader_election.cc:288] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:06:43.439767 4728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:43.441114 4728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [term 1 LEADER]: Becoming Leader. State: Replica: 7a82866733fa43d09abd13eedc5ee6df, State: Running, Role: LEADER
I20260806 04:06:43.441875 4728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a82866733fa43d09abd13eedc5ee6df" member_type: VOTER }
I20260806 04:06:43.442548 4725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:06:43.448531 4730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a82866733fa43d09abd13eedc5ee6df. Latest consensus state: current_term: 1 leader_uuid: "7a82866733fa43d09abd13eedc5ee6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a82866733fa43d09abd13eedc5ee6df" member_type: VOTER } }
I20260806 04:06:43.450382 4730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [sys.catalog]: This master's current role is: LEADER
I20260806 04:06:43.449663 4729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a82866733fa43d09abd13eedc5ee6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a82866733fa43d09abd13eedc5ee6df" member_type: VOTER } }
I20260806 04:06:43.451747 4729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df [sys.catalog]: This master's current role is: LEADER
I20260806 04:06:43.455585 4745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:06:43.460269 4745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:06:43.464198 3041 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 04:06:43.469449 4745 catalog_manager.cc:1347] Generated new cluster ID: 1480ea93da3342a797264138c21cde02
I20260806 04:06:43.469679 4745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:06:43.505149 4745 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:06:43.506806 4745 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:06:43.521757 4745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Generated new TSK 0
I20260806 04:06:43.522437 4745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:06:43.530953 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:43.537163 4756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:43.538861 4759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:43.540376 3041 server_base.cc:1034] running on GCE node
W20260806 04:06:43.540339 4761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:43.541599 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:43.541803 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:43.542008 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989203541976 us; error 0 us; skew 500 ppm
I20260806 04:06:43.542637 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:43.545410 3041 webserver.cc:460] Webserver started at http://127.2.248.65:36445/ using document root <none> and password file <none>
I20260806 04:06:43.545931 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:43.546126 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:43.546424 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:43.547657 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/instance:
uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.552573 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:06:43.556245 4767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.557102 3041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:06:43.557430 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root
uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.557812 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:43.567200 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:43.568567 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:43.570122 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:43.572510 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:43.572746 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.572994 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:43.573171 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.615922 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.65:43865
I20260806 04:06:43.616027 4853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.65:43865 every 8 connection(s)
I20260806 04:06:43.621050 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:43.629212 4862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:43.629539 4863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:43.632900 4868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:43.633320 3041 server_base.cc:1034] running on GCE node
I20260806 04:06:43.635600 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:43.635821 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:43.636032 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989203636016 us; error 0 us; skew 500 ppm
I20260806 04:06:43.636353 4856 heartbeater.cc:347] Connected to a master server at 127.2.248.126:44289
I20260806 04:06:43.636705 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:43.636715 4856 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:43.637591 4856 heartbeater.cc:511] Master 127.2.248.126:44289 requested a full tablet report, sending...
I20260806 04:06:43.639812 3041 webserver.cc:460] Webserver started at http://127.2.248.66:46361/ using document root <none> and password file <none>
I20260806 04:06:43.640244 4676 ts_manager.cc:194] Registered new tserver with Master: df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:06:43.640352 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:43.640707 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:43.640987 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:43.642280 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/instance:
uuid: "22fe557f852d4415a96fd683f970cc24"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.642449 4676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46886
I20260806 04:06:43.647258 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:06:43.650791 4875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.651551 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:06:43.651890 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root
uuid: "22fe557f852d4415a96fd683f970cc24"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.652184 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:43.672441 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:43.673766 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:43.675856 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:43.679036 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:43.679224 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.679509 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:43.679680 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.720641 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.66:34857
I20260806 04:06:43.720754 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.66:34857 every 8 connection(s)
I20260806 04:06:43.725780 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:43.732399 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:43.734876 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:43.738930 4967 heartbeater.cc:347] Connected to a master server at 127.2.248.126:44289
I20260806 04:06:43.739502 4967 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:43.740178 3041 server_base.cc:1034] running on GCE node
I20260806 04:06:43.740489 4967 heartbeater.cc:511] Master 127.2.248.126:44289 requested a full tablet report, sending...
W20260806 04:06:43.741743 4978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:43.742872 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:43.743072 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:43.743080 4676 ts_manager.cc:194] Registered new tserver with Master: 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:43.743382 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989203743362 us; error 0 us; skew 500 ppm
I20260806 04:06:43.744033 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:43.745340 4676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46900
I20260806 04:06:43.747020 3041 webserver.cc:460] Webserver started at http://127.2.248.67:40127/ using document root <none> and password file <none>
I20260806 04:06:43.747570 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:43.747781 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:43.748063 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:43.749315 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/instance:
uuid: "99690415a7b3441fbdb5e943667556a4"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.754361 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:06:43.757926 4990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.758886 3041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:06:43.759210 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root
uuid: "99690415a7b3441fbdb5e943667556a4"
format_stamp: "Formatted at 2026-08-06 04:06:43 on dist-test-slave-3kk6"
I20260806 04:06:43.759523 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:43.771620 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:43.772941 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:43.774467 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:43.776708 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:43.776947 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.777195 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:43.777377 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:43.818724 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.67:43701
I20260806 04:06:43.818892 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.67:43701 every 8 connection(s)
I20260806 04:06:43.834358 5071 heartbeater.cc:347] Connected to a master server at 127.2.248.126:44289
I20260806 04:06:43.834837 5071 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:43.835570 5071 heartbeater.cc:511] Master 127.2.248.126:44289 requested a full tablet report, sending...
I20260806 04:06:43.837604 4676 ts_manager.cc:194] Registered new tserver with Master: 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:43.837733 3041 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015911072s
I20260806 04:06:43.839962 4676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46902
I20260806 04:06:43.840437 3041 test_util.cc:268] Using random seed: 68242847
I20260806 04:06:43.866961 4676 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46908:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:06:43.873335 4676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:06:44.091835 5025 tablet_service.cc:1467] Processing CreateTablet for tablet 141037797aa84869bb31e99383874122 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 04:06:44.093375 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141037797aa84869bb31e99383874122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.095161 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 141037797aa84869bb31e99383874122 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 04:06:44.095983 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 57d20919f4994e71841cccb9546444a3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 04:06:44.096568 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141037797aa84869bb31e99383874122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.097349 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d20919f4994e71841cccb9546444a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.101418 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 8570417a88db412294c21c9108061c30 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 04:06:44.105415 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8570417a88db412294c21c9108061c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.107932 5024 tablet_service.cc:1467] Processing CreateTablet for tablet 57d20919f4994e71841cccb9546444a3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 04:06:44.109398 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d20919f4994e71841cccb9546444a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.109087 5023 tablet_service.cc:1467] Processing CreateTablet for tablet 8570417a88db412294c21c9108061c30 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 04:06:44.113054 5022 tablet_service.cc:1467] Processing CreateTablet for tablet 4fef94f36fa34d64a4485e7136f39f85 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 04:06:44.113364 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 30e5d6887268432aaf472e51f9ae0444 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 04:06:44.114460 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8570417a88db412294c21c9108061c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.116756 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e5d6887268432aaf472e51f9ae0444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.121029 5019 tablet_service.cc:1467] Processing CreateTablet for tablet 457d26714def4b128d6f433119c45d5f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 04:06:44.122423 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457d26714def4b128d6f433119c45d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.124439 5018 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 04:06:44.126552 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea51188c553493cbb9b842b0040bd4f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 04:06:44.131034 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea51188c553493cbb9b842b0040bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.133121 4803 tablet_service.cc:1467] Processing CreateTablet for tablet 141037797aa84869bb31e99383874122 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 04:06:44.135210 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.135891 4802 tablet_service.cc:1467] Processing CreateTablet for tablet 57d20919f4994e71841cccb9546444a3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 04:06:44.140865 4798 tablet_service.cc:1467] Processing CreateTablet for tablet 30e5d6887268432aaf472e51f9ae0444 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 04:06:44.142854 5017 tablet_service.cc:1467] Processing CreateTablet for tablet da2e93c822734c02a26703dabb801932 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 04:06:44.143335 4797 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea51188c553493cbb9b842b0040bd4f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 04:06:44.145568 4795 tablet_service.cc:1467] Processing CreateTablet for tablet 457d26714def4b128d6f433119c45d5f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 04:06:44.134495 4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141037797aa84869bb31e99383874122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.152458 4910 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 04:06:44.154280 4911 tablet_service.cc:1467] Processing CreateTablet for tablet 457d26714def4b128d6f433119c45d5f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 04:06:44.167853 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457d26714def4b128d6f433119c45d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.156127 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 4219680d51af4d0587436328294ba316 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 04:06:44.169772 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4219680d51af4d0587436328294ba316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.171355 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.157948 4909 tablet_service.cc:1467] Processing CreateTablet for tablet da2e93c822734c02a26703dabb801932 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 04:06:44.174266 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2e93c822734c02a26703dabb801932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.159797 4794 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 04:06:44.161854 4913 tablet_service.cc:1467] Processing CreateTablet for tablet 30e5d6887268432aaf472e51f9ae0444 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 04:06:44.176671 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e5d6887268432aaf472e51f9ae0444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.178013 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457d26714def4b128d6f433119c45d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.178687 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.141362 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fef94f36fa34d64a4485e7136f39f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.164649 4912 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea51188c553493cbb9b842b0040bd4f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 04:06:44.185936 4793 tablet_service.cc:1467] Processing CreateTablet for tablet da2e93c822734c02a26703dabb801932 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 04:06:44.187110 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea51188c553493cbb9b842b0040bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.187199 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea51188c553493cbb9b842b0040bd4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.165441 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 4fef94f36fa34d64a4485e7136f39f85 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 04:06:44.138793 4800 tablet_service.cc:1467] Processing CreateTablet for tablet 8570417a88db412294c21c9108061c30 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 04:06:44.166163 4914 tablet_service.cc:1467] Processing CreateTablet for tablet 4fef94f36fa34d64a4485e7136f39f85 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 04:06:44.145916 5016 tablet_service.cc:1467] Processing CreateTablet for tablet 4219680d51af4d0587436328294ba316 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 04:06:44.188789 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 4219680d51af4d0587436328294ba316 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 04:06:44.188805 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fef94f36fa34d64a4485e7136f39f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.194509 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fef94f36fa34d64a4485e7136f39f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.207067 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2e93c822734c02a26703dabb801932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.211133 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4219680d51af4d0587436328294ba316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.211534 4918 tablet_service.cc:1467] Processing CreateTablet for tablet faa361eb5ab5463895c3f7af5e87237c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 04:06:44.212906 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa361eb5ab5463895c3f7af5e87237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.216104 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 50ac9e0fb0ae490c88fc2d1584718458 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 04:06:44.217392 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4219680d51af4d0587436328294ba316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.219686 5018 tablet_service.cc:1467] Processing CreateTablet for tablet faa361eb5ab5463895c3f7af5e87237c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 04:06:44.217509 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ac9e0fb0ae490c88fc2d1584718458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.227713 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa361eb5ab5463895c3f7af5e87237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.231191 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8570417a88db412294c21c9108061c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.226047 5019 tablet_service.cc:1467] Processing CreateTablet for tablet 50ac9e0fb0ae490c88fc2d1584718458 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 04:06:44.235087 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e5d6887268432aaf472e51f9ae0444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.237258 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2e93c822734c02a26703dabb801932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.244685 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ac9e0fb0ae490c88fc2d1584718458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.245498 4802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d20919f4994e71841cccb9546444a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.271827 5092 tablet_bootstrap.cc:492] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:44.317337 5023 tablet_service.cc:1467] Processing CreateTablet for tablet ebcff471d5ff446ba22c34ec3724dfeb (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 04:06:44.328366 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcff471d5ff446ba22c34ec3724dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.330029 5018 tablet_service.cc:1467] Processing CreateTablet for tablet eaa370887c344934b64e0b3f59c6a3b0 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 04:06:44.333829 4799 tablet_service.cc:1467] Processing CreateTablet for tablet faa361eb5ab5463895c3f7af5e87237c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 04:06:44.361505 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa361eb5ab5463895c3f7af5e87237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.362391 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa370887c344934b64e0b3f59c6a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.343408 4909 tablet_service.cc:1467] Processing CreateTablet for tablet 7659c18c5db84b729be8dce3bcb9dd32 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 04:06:44.350894 4797 tablet_service.cc:1467] Processing CreateTablet for tablet 50ac9e0fb0ae490c88fc2d1584718458 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 04:06:44.371580 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ac9e0fb0ae490c88fc2d1584718458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.357020 5092 tablet_bootstrap.cc:654] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:44.359885 4918 tablet_service.cc:1467] Processing CreateTablet for tablet eaa370887c344934b64e0b3f59c6a3b0 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 04:06:44.339785 4916 tablet_service.cc:1467] Processing CreateTablet for tablet ebcff471d5ff446ba22c34ec3724dfeb (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 04:06:44.371582 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659c18c5db84b729be8dce3bcb9dd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.388938 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1f802648ed4dccb7235c747e9b1553 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 04:06:44.391187 4795 tablet_service.cc:1467] Processing CreateTablet for tablet ebcff471d5ff446ba22c34ec3724dfeb (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 04:06:44.391453 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1f802648ed4dccb7235c747e9b1553 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 04:06:44.392546 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcff471d5ff446ba22c34ec3724dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.394024 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcff471d5ff446ba22c34ec3724dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.393172 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 98064c6d64f74659bfa102bd5a341635 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 04:06:44.395046 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1f802648ed4dccb7235c747e9b1553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.395648 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98064c6d64f74659bfa102bd5a341635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.392580 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1f802648ed4dccb7235c747e9b1553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.395768 4912 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff5845199524dcb8deb5af416122df3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 04:06:44.396824 5097 tablet_bootstrap.cc:492] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:44.397017 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff5845199524dcb8deb5af416122df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.399408 5093 tablet_bootstrap.cc:492] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:44.382411 5025 tablet_service.cc:1467] Processing CreateTablet for tablet 7659c18c5db84b729be8dce3bcb9dd32 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 04:06:44.407990 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659c18c5db84b729be8dce3bcb9dd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.422613 5019 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff5845199524dcb8deb5af416122df3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 04:06:44.423925 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff5845199524dcb8deb5af416122df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.434894 5024 tablet_service.cc:1467] Processing CreateTablet for tablet 086f8af9433f44aa8aa4fccbfa90eb02 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 04:06:44.436197 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086f8af9433f44aa8aa4fccbfa90eb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.437225 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 086f8af9433f44aa8aa4fccbfa90eb02 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 04:06:44.441587 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086f8af9433f44aa8aa4fccbfa90eb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.421878 4803 tablet_service.cc:1467] Processing CreateTablet for tablet 7659c18c5db84b729be8dce3bcb9dd32 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 04:06:44.447733 4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659c18c5db84b729be8dce3bcb9dd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.426163 4794 tablet_service.cc:1467] Processing CreateTablet for tablet eaa370887c344934b64e0b3f59c6a3b0 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 04:06:44.452042 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa370887c344934b64e0b3f59c6a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:06:44.459478 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:44.463791 4914 tablet_service.cc:1467] Processing CreateTablet for tablet cf27da0cc45d4501b4dac0fbb6fedc16 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 04:06:44.465667 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf27da0cc45d4501b4dac0fbb6fedc16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.476953 5022 tablet_service.cc:1467] Processing CreateTablet for tablet 98064c6d64f74659bfa102bd5a341635 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 04:06:44.478108 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa370887c344934b64e0b3f59c6a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.479629 5097 tablet_bootstrap.cc:654] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:44.487831 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 874b55144315439cb842845d56ccb153 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 04:06:44.493232 5021 tablet_service.cc:1467] Processing CreateTablet for tablet cf27da0cc45d4501b4dac0fbb6fedc16 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 04:06:44.489368 5093 tablet_bootstrap.cc:654] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:44.492470 5016 tablet_service.cc:1467] Processing CreateTablet for tablet 27545c8363cd404189c9e93f3bd1426c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 04:06:44.502764 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874b55144315439cb842845d56ccb153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.499987 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98064c6d64f74659bfa102bd5a341635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.511840 4799 tablet_service.cc:1467] Processing CreateTablet for tablet cf27da0cc45d4501b4dac0fbb6fedc16 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 04:06:44.513496 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf27da0cc45d4501b4dac0fbb6fedc16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.515448 4910 tablet_service.cc:1467] Processing CreateTablet for tablet 654e437ca202414286305ce8a8c5c4c2 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 04:06:44.516752 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654e437ca202414286305ce8a8c5c4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.516558 4798 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1f802648ed4dccb7235c747e9b1553 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
W20260806 04:06:44.522841 4651 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 48139.5us
I20260806 04:06:44.523300 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27545c8363cd404189c9e93f3bd1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.527345 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1f802648ed4dccb7235c747e9b1553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.531319 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf27da0cc45d4501b4dac0fbb6fedc16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.536670 4800 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff5845199524dcb8deb5af416122df3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 04:06:44.536549 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 96c2cb602ae44188a8a52e2cecd5a807 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 04:06:44.542387 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c2cb602ae44188a8a52e2cecd5a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.506093 4913 tablet_service.cc:1467] Processing CreateTablet for tablet 8109ccbed70f46e0932c03eaddc3caf7 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 04:06:44.553377 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff5845199524dcb8deb5af416122df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.566500 5092 tablet_bootstrap.cc:492] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:44.567149 5092 ts_tablet_manager.cc:1397] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.296s user 0.060s sys 0.048s
I20260806 04:06:44.569200 4911 tablet_service.cc:1467] Processing CreateTablet for tablet cdc3f501bed64af09b04782801118ec3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 04:06:44.568938 5093 tablet_bootstrap.cc:492] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:44.570150 5093 ts_tablet_manager.cc:1397] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.171s user 0.024s sys 0.043s
I20260806 04:06:44.579641 5093 raft_consensus.cc:348] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.580260 5097 tablet_bootstrap.cc:492] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:44.580542 5093 raft_consensus.cc:374] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:44.580761 5097 ts_tablet_manager.cc:1397] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.184s user 0.014s sys 0.006s
I20260806 04:06:44.580933 5093 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:44.581853 5093 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.572968 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 27545c8363cd404189c9e93f3bd1426c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 04:06:44.569077 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 086f8af9433f44aa8aa4fccbfa90eb02 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 04:06:44.583988 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086f8af9433f44aa8aa4fccbfa90eb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.572666 5092 raft_consensus.cc:348] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.586817 5092 raft_consensus.cc:374] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:44.587145 5092 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:44.587867 5092 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.589162 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8109ccbed70f46e0932c03eaddc3caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.594786 5018 tablet_service.cc:1467] Processing CreateTablet for tablet 654e437ca202414286305ce8a8c5c4c2 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 04:06:44.602389 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc3f501bed64af09b04782801118ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.613262 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 96c2cb602ae44188a8a52e2cecd5a807 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 04:06:44.614583 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c2cb602ae44188a8a52e2cecd5a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.637260 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27545c8363cd404189c9e93f3bd1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.642347 4967 heartbeater.cc:503] Master 127.2.248.126:44289 was elected leader, sending a full tablet report...
I20260806 04:06:44.642880 5097 raft_consensus.cc:348] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.643678 5097 raft_consensus.cc:374] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:44.643970 5097 raft_consensus.cc:729] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:44.644649 5097 consensus_queue.cc:260] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.645694 4856 heartbeater.cc:503] Master 127.2.248.126:44289 was elected leader, sending a full tablet report...
I20260806 04:06:44.645871 5093 ts_tablet_manager.cc:1428] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.069s user 0.009s sys 0.014s
I20260806 04:06:44.647750 5092 ts_tablet_manager.cc:1428] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.078s user 0.029s sys 0.013s
I20260806 04:06:44.652781 5071 heartbeater.cc:503] Master 127.2.248.126:44289 was elected leader, sending a full tablet report...
I20260806 04:06:44.649950 5093 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:44.637910 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654e437ca202414286305ce8a8c5c4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.674160 4798 tablet_service.cc:1467] Processing CreateTablet for tablet 96c2cb602ae44188a8a52e2cecd5a807 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 04:06:44.678593 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 27545c8363cd404189c9e93f3bd1426c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 04:06:44.678897 4908 tablet_service.cc:1467] Processing CreateTablet for tablet b5088de56cbc4281b755364a8fddd084 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 04:06:44.680053 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5088de56cbc4281b755364a8fddd084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.683293 4912 tablet_service.cc:1467] Processing CreateTablet for tablet f8214005d1a744f9b566b933df1cfe4a (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 04:06:44.684043 4802 tablet_service.cc:1467] Processing CreateTablet for tablet b5088de56cbc4281b755364a8fddd084 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 04:06:44.684535 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8214005d1a744f9b566b933df1cfe4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.685353 4794 tablet_service.cc:1467] Processing CreateTablet for tablet cdc3f501bed64af09b04782801118ec3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 04:06:44.688886 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c2cb602ae44188a8a52e2cecd5a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.691126 5092 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:44.693480 5025 tablet_service.cc:1467] Processing CreateTablet for tablet b5088de56cbc4281b755364a8fddd084 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 04:06:44.696481 5019 tablet_service.cc:1467] Processing CreateTablet for tablet cdc3f501bed64af09b04782801118ec3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 04:06:44.702942 4797 tablet_service.cc:1467] Processing CreateTablet for tablet 654e437ca202414286305ce8a8c5c4c2 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 04:06:44.707139 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1ae23ce1104b3688f3f881c64ed473 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 04:06:44.712510 5024 tablet_service.cc:1467] Processing CreateTablet for tablet f8214005d1a744f9b566b933df1cfe4a (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 04:06:44.717321 4911 tablet_service.cc:1467] Processing CreateTablet for tablet a3ae3b855e0d4f98a67493c708f71d33 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 04:06:44.718938 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 98064c6d64f74659bfa102bd5a341635 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 04:06:44.720760 4914 tablet_service.cc:1467] Processing CreateTablet for tablet 77a878f4d22e44a290e162038ddd83e1 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 04:06:44.722136 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27545c8363cd404189c9e93f3bd1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.723618 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 8109ccbed70f46e0932c03eaddc3caf7 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 04:06:44.713487 5092 tablet_bootstrap.cc:654] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:44.726984 4802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5088de56cbc4281b755364a8fddd084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.727628 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc3f501bed64af09b04782801118ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.732290 5097 ts_tablet_manager.cc:1428] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.151s user 0.011s sys 0.001s
I20260806 04:06:44.733186 5093 tablet_bootstrap.cc:654] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:44.733367 5097 tablet_bootstrap.cc:492] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:44.723618 5022 tablet_service.cc:1467] Processing CreateTablet for tablet 8109ccbed70f46e0932c03eaddc3caf7 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 04:06:44.742782 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654e437ca202414286305ce8a8c5c4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.746516 4913 tablet_service.cc:1467] Processing CreateTablet for tablet 12fab895c8c34b9d82a95dc9f77980c6 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 04:06:44.746872 4910 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1ae23ce1104b3688f3f881c64ed473 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 04:06:44.738847 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ae3b855e0d4f98a67493c708f71d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.748056 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5088de56cbc4281b755364a8fddd084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.749503 5097 tablet_bootstrap.cc:654] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:44.750052 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1ae23ce1104b3688f3f881c64ed473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.750947 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98064c6d64f74659bfa102bd5a341635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.753221 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8109ccbed70f46e0932c03eaddc3caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.754236 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc3f501bed64af09b04782801118ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.759711 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8109ccbed70f46e0932c03eaddc3caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.742036 4800 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1ae23ce1104b3688f3f881c64ed473 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 04:06:44.760788 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1ae23ce1104b3688f3f881c64ed473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.761282 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8214005d1a744f9b566b933df1cfe4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.764789 4803 tablet_service.cc:1467] Processing CreateTablet for tablet f8214005d1a744f9b566b933df1cfe4a (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 04:06:44.738581 5023 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0f147300804e5fab336f0242e01ec4 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 04:06:44.767040 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12fab895c8c34b9d82a95dc9f77980c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.767911 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0f147300804e5fab336f0242e01ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.766119 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 874b55144315439cb842845d56ccb153 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 04:06:44.769001 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1ae23ce1104b3688f3f881c64ed473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.769843 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874b55144315439cb842845d56ccb153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.706362 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0f147300804e5fab336f0242e01ec4 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 04:06:44.771062 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0f147300804e5fab336f0242e01ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.778730 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a878f4d22e44a290e162038ddd83e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.794579 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 21b536e5a86f4281a4d9ee87a11d409f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 04:06:44.795966 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b536e5a86f4281a4d9ee87a11d409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.802450 4909 tablet_service.cc:1467] Processing CreateTablet for tablet 54785970285b4054b83ec528e156c25c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 04:06:44.803697 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54785970285b4054b83ec528e156c25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.812064 4912 tablet_service.cc:1467] Processing CreateTablet for tablet 589278a46368433c87a150021544ba2e (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 04:06:44.813359 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589278a46368433c87a150021544ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.813045 5018 tablet_service.cc:1467] Processing CreateTablet for tablet a3ae3b855e0d4f98a67493c708f71d33 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 04:06:44.814229 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ae3b855e0d4f98a67493c708f71d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.810446 4798 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0f147300804e5fab336f0242e01ec4 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 04:06:44.824941 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0f147300804e5fab336f0242e01ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.826598 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4bc22d4f8c4b178aca679682ba4b64 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 04:06:44.764802 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 28c567d00bdc4e3b800967318816ba4d (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 04:06:44.856348 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c567d00bdc4e3b800967318816ba4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.831132 4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8214005d1a744f9b566b933df1cfe4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.832288 5097 tablet_bootstrap.cc:492] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:44.832500 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 12fab895c8c34b9d82a95dc9f77980c6 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 04:06:44.833024 5103 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:44.843039 4795 tablet_service.cc:1467] Processing CreateTablet for tablet 77a878f4d22e44a290e162038ddd83e1 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 04:06:44.798156 5016 tablet_service.cc:1467] Processing CreateTablet for tablet 77a878f4d22e44a290e162038ddd83e1 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 04:06:44.851059 4800 tablet_service.cc:1467] Processing CreateTablet for tablet 874b55144315439cb842845d56ccb153 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 04:06:44.862026 5103 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.863590 5092 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:44.864135 5092 ts_tablet_manager.cc:1397] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.173s user 0.008s sys 0.011s
I20260806 04:06:44.864529 5103 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:44.865516 5097 ts_tablet_manager.cc:1397] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.132s user 0.014s sys 0.000s
I20260806 04:06:44.888298 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12fab895c8c34b9d82a95dc9f77980c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.874637 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874b55144315439cb842845d56ccb153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.877411 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4bc22d4f8c4b178aca679682ba4b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.893169 5092 raft_consensus.cc:348] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.893937 5092 raft_consensus.cc:374] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:44.894248 5092 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:44.894969 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a878f4d22e44a290e162038ddd83e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.900960 4798 tablet_service.cc:1467] Processing CreateTablet for tablet a3ae3b855e0d4f98a67493c708f71d33 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 04:06:44.902280 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ae3b855e0d4f98a67493c708f71d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.904305 5022 tablet_service.cc:1467] Processing CreateTablet for tablet 28c567d00bdc4e3b800967318816ba4d (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 04:06:44.921947 5105 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:44.922465 5105 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.923957 4911 tablet_service.cc:1467] Processing CreateTablet for tablet a0ff8f19c60b43b99aa42dd6c9cee458 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 04:06:44.925380 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ff8f19c60b43b99aa42dd6c9cee458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.908103 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 12fab895c8c34b9d82a95dc9f77980c6 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 04:06:44.909474 5093 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:44.931119 5093 ts_tablet_manager.cc:1397] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.282s user 0.016s sys 0.000s
I20260806 04:06:44.931350 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12fab895c8c34b9d82a95dc9f77980c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.913298 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a878f4d22e44a290e162038ddd83e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.913883 5097 raft_consensus.cc:348] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.925489 5105 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:44.907727 4912 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd77d1ae4fc468f8a9d461acb07056b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 04:06:44.939910 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd77d1ae4fc468f8a9d461acb07056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.934106 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 0940457704114112b22dab24a18b43b6 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 04:06:44.949651 5023 tablet_service.cc:1467] Processing CreateTablet for tablet 21b536e5a86f4281a4d9ee87a11d409f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 04:06:44.951287 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 54785970285b4054b83ec528e156c25c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 04:06:44.957194 5019 tablet_service.cc:1467] Processing CreateTablet for tablet 589278a46368433c87a150021544ba2e (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 04:06:44.940752 5097 raft_consensus.cc:374] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:44.962687 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c567d00bdc4e3b800967318816ba4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.969604 5097 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:44.973600 4802 tablet_service.cc:1467] Processing CreateTablet for tablet 21b536e5a86f4281a4d9ee87a11d409f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 04:06:44.974993 4802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b536e5a86f4281a4d9ee87a11d409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.973649 5097 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.981303 5097 ts_tablet_manager.cc:1428] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.109s user 0.005s sys 0.001s
I20260806 04:06:44.982286 5097 tablet_bootstrap.cc:492] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:44.983449 5093 raft_consensus.cc:348] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.984234 5093 raft_consensus.cc:374] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:44.984540 5093 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:44.985337 5093 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.961786 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 54785970285b4054b83ec528e156c25c (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 04:06:44.989707 4797 tablet_service.cc:1467] Processing CreateTablet for tablet 589278a46368433c87a150021544ba2e (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 04:06:44.986402 4914 tablet_service.cc:1467] Processing CreateTablet for tablet df7a93f69159461280c8b02b8ff62358 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 04:06:44.971463 5092 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:44.997802 5104 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.003499 5104 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.003962 5025 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4bc22d4f8c4b178aca679682ba4b64 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 04:06:45.006836 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54785970285b4054b83ec528e156c25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.008056 4915 tablet_service.cc:1467] Processing CreateTablet for tablet f75362215d584713a892bd7319a22469 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 04:06:45.008581 5103 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.009059 5103 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.011178 5103 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.012140 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.015743 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.014815 4798 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4bc22d4f8c4b178aca679682ba4b64 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 04:06:45.019341 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0940457704114112b22dab24a18b43b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:44.993721 4913 tablet_service.cc:1467] Processing CreateTablet for tablet a4bb640df6c2435ea743b7157dd130bd (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
W20260806 04:06:45.019716 4882 leader_election.cc:341] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:45.025276 4882 leader_election.cc:341] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.026013 4800 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd77d1ae4fc468f8a9d461acb07056b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 04:06:45.035173 4795 tablet_service.cc:1467] Processing CreateTablet for tablet a0ff8f19c60b43b99aa42dd6c9cee458 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 04:06:45.017573 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b536e5a86f4281a4d9ee87a11d409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.044355 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589278a46368433c87a150021544ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.063885 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54785970285b4054b83ec528e156c25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.067101 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7a93f69159461280c8b02b8ff62358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.075371 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4bc22d4f8c4b178aca679682ba4b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.083103 4793 tablet_service.cc:1467] Processing CreateTablet for tablet a4bb640df6c2435ea743b7157dd130bd (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 04:06:45.084097 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd77d1ae4fc468f8a9d461acb07056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.076215 4794 tablet_service.cc:1467] Processing CreateTablet for tablet 28c567d00bdc4e3b800967318816ba4d (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 04:06:45.084864 5093 ts_tablet_manager.cc:1428] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.153s user 0.007s sys 0.001s
I20260806 04:06:45.085542 5018 tablet_service.cc:1467] Processing CreateTablet for tablet a4bb640df6c2435ea743b7157dd130bd (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 04:06:45.086571 4803 tablet_service.cc:1467] Processing CreateTablet for tablet 0940457704114112b22dab24a18b43b6 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 04:06:45.087082 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4bc22d4f8c4b178aca679682ba4b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.089174 4911 tablet_service.cc:1467] Processing CreateTablet for tablet b75ec76cc4cd43bc90264007dff133f3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 04:06:45.091115 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589278a46368433c87a150021544ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.092885 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4bb640df6c2435ea743b7157dd130bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.083478 5122 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.092983 4908 tablet_service.cc:1467] Processing CreateTablet for tablet fdc0072b6de7424caf2520a26bdcd0b0 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 04:06:45.098495 5024 tablet_service.cc:1467] Processing CreateTablet for tablet a0ff8f19c60b43b99aa42dd6c9cee458 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 04:06:45.096566 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd77d1ae4fc468f8a9d461acb07056b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 04:06:45.093999 5122 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.099874 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ff8f19c60b43b99aa42dd6c9cee458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.104322 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 0940457704114112b22dab24a18b43b6 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 04:06:45.106019 5104 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:45.105082 4910 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb317a04be74515913f7aee753c193b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 04:06:45.113219 5105 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.113864 5105 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.115995 5105 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.117846 5122 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:45.127008 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75362215d584713a892bd7319a22469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.127444 5093 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:45.131008 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a7d4c7807476384d5ce509d4029f9 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 04:06:45.151355 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd77d1ae4fc468f8a9d461acb07056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.151546 5097 tablet_bootstrap.cc:654] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.156826 5023 tablet_service.cc:1467] Processing CreateTablet for tablet df7a93f69159461280c8b02b8ff62358 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 04:06:45.159319 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ff8f19c60b43b99aa42dd6c9cee458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.160571 4799 tablet_service.cc:1467] Processing CreateTablet for tablet df7a93f69159461280c8b02b8ff62358 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 04:06:45.162258 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7a93f69159461280c8b02b8ff62358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.171556 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7a93f69159461280c8b02b8ff62358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.179612 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c567d00bdc4e3b800967318816ba4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.180191 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0940457704114112b22dab24a18b43b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.180652 4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0940457704114112b22dab24a18b43b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.177510 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4bb640df6c2435ea743b7157dd130bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.191213 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4bb640df6c2435ea743b7157dd130bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.192091 4797 tablet_service.cc:1467] Processing CreateTablet for tablet f75362215d584713a892bd7319a22469 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 04:06:45.193419 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75362215d584713a892bd7319a22469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.194180 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b75ec76cc4cd43bc90264007dff133f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.196992 4909 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2ae1bef38445b908545a06111f237 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 04:06:45.198277 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2ae1bef38445b908545a06111f237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.208396 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc0072b6de7424caf2520a26bdcd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.211658 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:45.213829 5092 ts_tablet_manager.cc:1428] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.348s user 0.016s sys 0.014s
W20260806 04:06:45.214149 4768 leader_election.cc:341] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.215154 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:45.216581 4768 leader_election.cc:341] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.218950 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a7d4c7807476384d5ce509d4029f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.235718 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.236483 5038 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:45.238034 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.239421 4876 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.240293 5103 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.240648 5103 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.240984 5103 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:45.241289 4876 leader_election.cc:341] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.241377 5093 tablet_bootstrap.cc:654] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.241816 4876 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.250876 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb317a04be74515913f7aee753c193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.251814 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.253043 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.256119 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.256657 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.255915 5018 tablet_service.cc:1467] Processing CreateTablet for tablet f75362215d584713a892bd7319a22469 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 04:06:45.258746 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75362215d584713a892bd7319a22469. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:06:45.263890 4996 leader_election.cc:341] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:45.264690 4996 leader_election.cc:341] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.267023 5092 tablet_bootstrap.cc:492] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
W20260806 04:06:45.268980 4768 leader_election.cc:341] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.269419 4768 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:45.270628 5105 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.278900 4798 tablet_service.cc:1467] Processing CreateTablet for tablet fdc0072b6de7424caf2520a26bdcd0b0 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 04:06:45.280257 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc0072b6de7424caf2520a26bdcd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.280771 4800 tablet_service.cc:1467] Processing CreateTablet for tablet b75ec76cc4cd43bc90264007dff133f3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 04:06:45.281005 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb317a04be74515913f7aee753c193b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 04:06:45.282207 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b75ec76cc4cd43bc90264007dff133f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.279870 5016 tablet_service.cc:1467] Processing CreateTablet for tablet fdc0072b6de7424caf2520a26bdcd0b0 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 04:06:45.283880 5097 tablet_bootstrap.cc:492] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:45.289597 4802 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a7d4c7807476384d5ce509d4029f9 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260806 04:06:45.290413 4768 leader_election.cc:341] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.291585 4914 tablet_service.cc:1467] Processing CreateTablet for tablet 228767bbb3b643c9977ce6fc5b074827 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 04:06:45.294464 5025 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb317a04be74515913f7aee753c193b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 04:06:45.307238 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:45.250169 5103 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.288661 5024 tablet_service.cc:1467] Processing CreateTablet for tablet b75ec76cc4cd43bc90264007dff133f3 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 04:06:45.305531 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc0072b6de7424caf2520a26bdcd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.274353 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 01722f8c013b4c0fb8ae76f83562ca5f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 04:06:45.306900 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb317a04be74515913f7aee753c193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.304242 4912 tablet_service.cc:1467] Processing CreateTablet for tablet f573a65ecea34309b58a81b5fcbfdc02 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 04:06:45.314131 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb317a04be74515913f7aee753c193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.314548 5103 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.311231 5105 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.306922 5092 tablet_bootstrap.cc:654] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.324600 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a7d4c7807476384d5ce509d4029f9 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 04:06:45.325749 5022 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2ae1bef38445b908545a06111f237 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 04:06:45.323536 5103 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.320204 5097 ts_tablet_manager.cc:1397] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.338s user 0.017s sys 0.016s
I20260806 04:06:45.327978 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228767bbb3b643c9977ce6fc5b074827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.328442 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:06:45.329934 4882 leader_election.cc:341] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.329833 5097 raft_consensus.cc:348] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.330677 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f573a65ecea34309b58a81b5fcbfdc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.332050 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01722f8c013b4c0fb8ae76f83562ca5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.339078 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b75ec76cc4cd43bc90264007dff133f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.339687 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a7d4c7807476384d5ce509d4029f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.306334 4803 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2ae1bef38445b908545a06111f237 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 04:06:45.345795 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 228767bbb3b643c9977ce6fc5b074827 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 04:06:45.347057 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2ae1bef38445b908545a06111f237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.351013 4913 tablet_service.cc:1467] Processing CreateTablet for tablet cfd13a1778034c599358d37b5d72c374 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 04:06:45.351450 4802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a7d4c7807476384d5ce509d4029f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.352401 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd13a1778034c599358d37b5d72c374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.353623 4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2ae1bef38445b908545a06111f237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.334444 4795 tablet_service.cc:1467] Processing CreateTablet for tablet 01722f8c013b4c0fb8ae76f83562ca5f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 04:06:45.357647 5097 raft_consensus.cc:374] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.359591 5097 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.338419 4799 tablet_service.cc:1467] Processing CreateTablet for tablet f573a65ecea34309b58a81b5fcbfdc02 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 04:06:45.360051 5023 tablet_service.cc:1467] Processing CreateTablet for tablet 228767bbb3b643c9977ce6fc5b074827 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 04:06:45.351068 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 793f3bf4a49a4d108609efd686fb6c85 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 04:06:45.366312 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228767bbb3b643c9977ce6fc5b074827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.366643 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f3bf4a49a4d108609efd686fb6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.377084 5092 tablet_bootstrap.cc:492] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:45.377678 5092 ts_tablet_manager.cc:1397] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.014s
I20260806 04:06:45.380466 5092 raft_consensus.cc:348] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.381256 5092 raft_consensus.cc:374] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.381592 5092 raft_consensus.cc:729] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.382421 5092 consensus_queue.cc:260] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.384625 5092 ts_tablet_manager.cc:1428] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260806 04:06:45.364802 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 123064us
I20260806 04:06:45.392933 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:45.393524 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:45.394362 4930 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:06:45.394673 4992 leader_election.cc:341] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.395110 4992 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.397372 5122 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.398540 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01722f8c013b4c0fb8ae76f83562ca5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.399394 4794 tablet_service.cc:1467] Processing CreateTablet for tablet 793f3bf4a49a4d108609efd686fb6c85 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 04:06:45.400597 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f3bf4a49a4d108609efd686fb6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.404891 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:45.405493 5037 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.407083 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228767bbb3b643c9977ce6fc5b074827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.408068 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 01722f8c013b4c0fb8ae76f83562ca5f (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 04:06:45.409325 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01722f8c013b4c0fb8ae76f83562ca5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.410836 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 5854d52dcd124250a917b57b14dd2afc (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 04:06:45.419517 4992 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.419837 4800 tablet_service.cc:1467] Processing CreateTablet for tablet 997e4d9ff3ca465199a888e8af23ed7b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 04:06:45.421980 4800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997e4d9ff3ca465199a888e8af23ed7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.424127 5092 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:45.425426 4911 tablet_service.cc:1467] Processing CreateTablet for tablet a80ee0f1cc7747e8b736886e63200f37 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 04:06:45.427573 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f573a65ecea34309b58a81b5fcbfdc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.428350 4912 tablet_service.cc:1467] Processing CreateTablet for tablet 997e4d9ff3ca465199a888e8af23ed7b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 04:06:45.431411 5092 tablet_bootstrap.cc:654] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.432641 5037 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:45.432452 5017 tablet_service.cc:1467] Processing CreateTablet for tablet cfd13a1778034c599358d37b5d72c374 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 04:06:45.433204 5122 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.436931 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5854d52dcd124250a917b57b14dd2afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.428527 5022 tablet_service.cc:1467] Processing CreateTablet for tablet f573a65ecea34309b58a81b5fcbfdc02 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 04:06:45.433943 4876 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.441670 5019 tablet_service.cc:1467] Processing CreateTablet for tablet 793f3bf4a49a4d108609efd686fb6c85 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 04:06:45.442740 5129 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:45.443104 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f3bf4a49a4d108609efd686fb6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.443187 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80ee0f1cc7747e8b736886e63200f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.444043 5129 raft_consensus.cc:686] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:45.444409 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f573a65ecea34309b58a81b5fcbfdc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.444960 5129 consensus_queue.cc:237] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.447268 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd13a1778034c599358d37b5d72c374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.395898 5097 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.462852 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997e4d9ff3ca465199a888e8af23ed7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:06:45.463028 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:45.451260 4797 tablet_service.cc:1467] Processing CreateTablet for tablet cfd13a1778034c599358d37b5d72c374 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 04:06:45.465493 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd13a1778034c599358d37b5d72c374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.449342 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 5666124a588f45948039fcd27abc9cf4 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 04:06:45.460341 5097 ts_tablet_manager.cc:1428] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.133s user 0.007s sys 0.000s
I20260806 04:06:45.469238 5122 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.469717 5122 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.472617 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.473295 4818 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:45.474493 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:45.475052 4908 tablet_service.cc:1467] Processing CreateTablet for tablet 78dcfac52d99420a82f5c7b402a782d7 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 04:06:45.475577 5123 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.476027 5097 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:45.475979 5123 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.476854 5123 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.476441 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5666124a588f45948039fcd27abc9cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.479889 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:45.482365 4992 leader_election.cc:341] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.472956 5093 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:45.480353 5122 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:45.483412 5093 ts_tablet_manager.cc:1397] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.356s user 0.012s sys 0.021s
I20260806 04:06:45.489681 5093 raft_consensus.cc:348] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.490403 5093 raft_consensus.cc:374] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.490864 5093 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.491837 5025 tablet_service.cc:1467] Processing CreateTablet for tablet 5666124a588f45948039fcd27abc9cf4 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 04:06:45.493113 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5666124a588f45948039fcd27abc9cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.451967 5018 tablet_service.cc:1467] Processing CreateTablet for tablet 997e4d9ff3ca465199a888e8af23ed7b (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 04:06:45.491863 5093 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.495563 5020 tablet_service.cc:1467] Processing CreateTablet for tablet a80ee0f1cc7747e8b736886e63200f37 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 04:06:45.495997 5018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997e4d9ff3ca465199a888e8af23ed7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.485445 5123 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.459867 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 5854d52dcd124250a917b57b14dd2afc (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 04:06:45.498991 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dcfac52d99420a82f5c7b402a782d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.510149 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80ee0f1cc7747e8b736886e63200f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.518985 5105 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.519527 5105 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.521487 5105 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.521771 4794 tablet_service.cc:1467] Processing CreateTablet for tablet 5854d52dcd124250a917b57b14dd2afc (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 04:06:45.523890 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:45.524124 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.524727 5123 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:06:45.525323 4768 leader_election.cc:341] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.502576 4798 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 44) took 1026 ms. Trace:
W20260806 04:06:45.526011 4768 leader_election.cc:341] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.526438 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:45.526624 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5854d52dcd124250a917b57b14dd2afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.526239 4798 rpcz_store.cc:276] 0806 04:06:44.476185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.277874 (+801689us) service_pool.cc:227] Handling call
0806 04:06:45.279780 (+ 1906us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.279880 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.445652 (+165772us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.502053 (+ 56401us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.502514 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.528729 4798 tablet_service.cc:1467] Processing CreateTablet for tablet a80ee0f1cc7747e8b736886e63200f37 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 04:06:45.529831 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:06:45.531179 4992 leader_election.cc:341] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.531831 5105 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.545115 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 78dcfac52d99420a82f5c7b402a782d7 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 04:06:45.504297 5093 ts_tablet_manager.cc:1428] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260806 04:06:45.547247 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dcfac52d99420a82f5c7b402a782d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.553380 5016 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34040 (request call id 44) took 1025 ms. Trace:
I20260806 04:06:45.553932 5016 rpcz_store.cc:276] 0806 04:06:44.527966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.278986 (+751020us) service_pool.cc:227] Handling call
0806 04:06:45.305053 (+ 26067us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.305154 (+ 101us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.376963 (+ 71809us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.547525 (+170562us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.553317 (+ 5792us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.556170 5092 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:45.557972 5092 ts_tablet_manager.cc:1397] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.017s
I20260806 04:06:45.556285 4674 catalog_manager.cc:5495] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:45.563987 5092 raft_consensus.cc:348] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.564867 5092 raft_consensus.cc:374] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.565305 5092 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.566035 5092 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.548300 5024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34040 (request call id 45) took 1016 ms. Trace:
I20260806 04:06:45.566917 5024 rpcz_store.cc:276] 0806 04:06:44.531530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.287812 (+756282us) service_pool.cc:227] Handling call
0806 04:06:45.323436 (+ 35624us) spinlock_profiling.cc:243] Waited 7.65 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de908d21 00007f23de8e61e3 00007f23de897869 00007f23d7e99c44 00007f23d7e99bd2 00007f23d7e99b01
0806 04:06:45.323586 (+ 150us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.323707 (+ 121us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.373720 (+ 50013us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.477263 (+103543us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.548221 (+ 70958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17588480}
I20260806 04:06:45.568650 5092 ts_tablet_manager.cc:1428] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260806 04:06:45.580572 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:45.581276 4818 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.585183 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80ee0f1cc7747e8b736886e63200f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.587493 4799 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 50) took 1034 ms. Trace:
I20260806 04:06:45.587939 4799 rpcz_store.cc:276] 0806 04:06:44.553117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.337502 (+784385us) service_pool.cc:227] Handling call
0806 04:06:45.427025 (+ 89523us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fae82 00007f23df9138b9 00007f23de8265d8 00007f23de8e62a0 00007f23de897869 00007f23d7e99c44 00007f23d7e99bd2 00007f23d7e99b01 00007f23d7e99a7c
0806 04:06:45.427095 (+ 70us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.427190 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.562676 (+135486us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.586886 (+ 24210us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.587416 (+ 530us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41092864}
I20260806 04:06:45.590874 5129 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.590715 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 78dcfac52d99420a82f5c7b402a782d7 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 04:06:45.591416 5093 tablet_bootstrap.cc:492] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:45.591491 5129 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.595585 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.596421 5037 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:45.599709 4876 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:45.600811 5103 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.601243 5103 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.601854 5103 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.592344 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dcfac52d99420a82f5c7b402a782d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.605852 4795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 49) took 1066 ms. Trace:
I20260806 04:06:45.606597 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5854d52dcd124250a917b57b14dd2afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.606287 4795 rpcz_store.cc:276] 0806 04:06:44.539347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.333552 (+794205us) service_pool.cc:227] Handling call
0806 04:06:45.398054 (+ 64502us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.398167 (+ 113us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.483189 (+ 85022us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.545962 (+ 62773us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.605787 (+ 59825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.609192 4791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 46) took 1123 ms. Trace:
I20260806 04:06:45.609699 4791 rpcz_store.cc:276] 0806 04:06:44.485404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.280198 (+794794us) service_pool.cc:227] Handling call
0806 04:06:45.282546 (+ 2348us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.282636 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.477695 (+195059us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.546191 (+ 68496us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.609125 (+ 62934us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1898752}
I20260806 04:06:45.610832 5129 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:06:45.612910 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:45.614586 4882 leader_election.cc:341] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.543841 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 5666124a588f45948039fcd27abc9cf4 (DEFAULT_TABLE table=test-workload [id=0735bc8993c646e89c81220829e37a88]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 04:06:45.622453 5103 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.622910 4803 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 48) took 1129 ms. Trace:
I20260806 04:06:45.632009 4803 rpcz_store.cc:276] 0806 04:06:44.493131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.305411 (+812280us) service_pool.cc:227] Handling call
0806 04:06:45.343170 (+ 37759us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.343293 (+ 123us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.503669 (+160376us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.572377 (+ 68708us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.622845 (+ 50468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.624346 5097 tablet_bootstrap.cc:654] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.635983 4802 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 47) took 1145 ms. Trace:
I20260806 04:06:45.615386 4818 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:45.627329 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5666124a588f45948039fcd27abc9cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:45.639508 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:45.631055 5092 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:45.640782 5123 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:45.641360 5123 raft_consensus.cc:686] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:45.642300 5123 consensus_queue.cc:237] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.636391 4802 rpcz_store.cc:276] 0806 04:06:44.490157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.285759 (+795602us) service_pool.cc:227] Handling call
0806 04:06:45.321908 (+ 36149us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.326783 (+ 4875us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.390249 (+ 63466us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.568520 (+178271us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.635920 (+ 67400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.626828 5093 tablet_bootstrap.cc:654] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.636498 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:45.650766 5037 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.651805 5103 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
W20260806 04:06:45.653025 5072 tablet.cc:2369] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:06:45.653412 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:06:45.666109 4882 leader_election.cc:341] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.661652 5037 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:45.667804 5092 tablet_bootstrap.cc:654] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.668838 4797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 52) took 1047 ms. Trace:
I20260806 04:06:45.668525 4876 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.669348 4797 rpcz_store.cc:276] 0806 04:06:44.621041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.364395 (+743354us) service_pool.cc:227] Handling call
0806 04:06:45.465005 (+100610us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.465116 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.590554 (+125438us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.642626 (+ 52072us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.668733 (+ 26107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25501}
I20260806 04:06:45.672328 5103 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:45.673395 5103 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:45.674516 5103 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.672377 4800 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 54) took 1003 ms. Trace:
I20260806 04:06:45.678845 4800 rpcz_store.cc:276] 0806 04:06:44.668704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.418975 (+750271us) service_pool.cc:227] Handling call
0806 04:06:45.421508 (+ 2533us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.421614 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.603468 (+181854us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.638110 (+ 34642us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.672298 (+ 34188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.684020 5019 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34040 (request call id 53) took 1006 ms. Trace:
I20260806 04:06:45.688771 5097 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:45.692035 5097 ts_tablet_manager.cc:1397] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.219s user 0.010s sys 0.006s
I20260806 04:06:45.690922 5019 rpcz_store.cc:276] 0806 04:06:44.677255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.440730 (+763475us) service_pool.cc:227] Handling call
0806 04:06:45.442587 (+ 1857us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.442731 (+ 144us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.653346 (+210615us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.683335 (+ 29989us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.683952 (+ 617us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.697681 4669 catalog_manager.cc:5495] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:45.701565 5097 raft_consensus.cc:348] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.702703 5097 raft_consensus.cc:374] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.703159 5097 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.703949 5092 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:45.710109 4793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33546 (request call id 57) took 1013 ms. Trace:
I20260806 04:06:45.704394 5097 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:45.713136 5092 ts_tablet_manager.cc:1397] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.082s user 0.015s sys 0.007s
I20260806 04:06:45.714486 5097 ts_tablet_manager.cc:1428] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260806 04:06:45.717509 5097 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:45.720041 5092 raft_consensus.cc:348] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.720870 5092 raft_consensus.cc:374] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.721184 5092 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.724439 5097 tablet_bootstrap.cc:654] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.711875 4793 rpcz_store.cc:276] 0806 04:06:44.696987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:06:45.542796 (+845809us) service_pool.cc:227] Handling call
0806 04:06:45.626826 (+ 84030us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:06:45.626945 (+ 119us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:06:45.662549 (+ 35604us) tablet_metadata.cc:641] Metadata flushed
0806 04:06:45.709309 (+ 46760us) tablet_replica.cc:205] Creating consensus instance
0806 04:06:45.710018 (+ 709us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:06:45.727953 5092 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.730501 5092 ts_tablet_manager.cc:1428] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 04:06:45.731597 5092 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:45.739727 5093 tablet_bootstrap.cc:492] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:45.740784 5093 ts_tablet_manager.cc:1397] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.150s user 0.016s sys 0.002s
I20260806 04:06:45.743577 5093 raft_consensus.cc:348] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.744379 5093 raft_consensus.cc:374] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.744627 5093 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.745569 5093 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.748020 5097 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:45.748569 5097 ts_tablet_manager.cc:1397] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.004s
I20260806 04:06:45.748569 5093 ts_tablet_manager.cc:1428] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:06:45.749804 5093 tablet_bootstrap.cc:492] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:45.751816 5097 raft_consensus.cc:348] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.752769 5097 raft_consensus.cc:374] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.753156 5097 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.754007 5097 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.757085 5097 ts_tablet_manager.cc:1428] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260806 04:06:45.758105 5097 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:45.758442 5093 tablet_bootstrap.cc:654] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.761626 5092 tablet_bootstrap.cc:654] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.767836 5097 tablet_bootstrap.cc:654] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.781811 5092 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:45.782101 5097 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:45.782397 5092 ts_tablet_manager.cc:1397] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.004s
I20260806 04:06:45.782840 5097 ts_tablet_manager.cc:1397] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.012s
I20260806 04:06:45.785917 4674 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:45.787001 5097 raft_consensus.cc:348] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.788321 5097 raft_consensus.cc:374] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.788758 5097 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.790156 5097 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.794622 5097 ts_tablet_manager.cc:1428] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260806 04:06:45.796193 5097 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:45.797310 5105 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.797958 5105 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.802433 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:45.804388 4768 leader_election.cc:341] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.805769 5105 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.807731 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.807664 5092 raft_consensus.cc:348] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.808593 5092 raft_consensus.cc:374] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.808655 5037 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:45.808941 5092 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.810520 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:45.810597 5092 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.811535 5105 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.812180 5105 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.812553 5105 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.814319 5092 ts_tablet_manager.cc:1428] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260806 04:06:45.815729 5092 tablet_bootstrap.cc:492] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:45.821141 5097 tablet_bootstrap.cc:654] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.826035 5093 tablet_bootstrap.cc:492] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:45.827000 5093 ts_tablet_manager.cc:1397] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.077s user 0.026s sys 0.035s
I20260806 04:06:45.827140 5092 tablet_bootstrap.cc:654] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.830066 5103 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.830592 5103 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.831928 5093 raft_consensus.cc:348] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.832964 5093 raft_consensus.cc:374] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.833314 5093 raft_consensus.cc:729] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.835537 5103 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.834407 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.834385 5093 consensus_queue.cc:260] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.837195 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.838416 5093 ts_tablet_manager.cc:1428] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20260806 04:06:45.838629 4876 leader_election.cc:341] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.834681 5105 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:45.839634 4882 leader_election.cc:341] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.840297 4882 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:45.841619 5103 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:45.842487 5097 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:45.842834 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:45.843042 5097 ts_tablet_manager.cc:1397] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.016s
I20260806 04:06:45.844089 5093 tablet_bootstrap.cc:492] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:45.844393 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:45.844658 4768 leader_election.cc:341] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.844985 5037 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.846769 5097 raft_consensus.cc:348] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.847635 5097 raft_consensus.cc:374] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.847961 5097 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.848762 5097 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.852658 5093 tablet_bootstrap.cc:654] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.855347 5037 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:45.855935 5105 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.867120 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:45.869091 5105 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:45.869848 5105 raft_consensus.cc:686] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:45.870904 5092 tablet_bootstrap.cc:492] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:45.871661 5092 ts_tablet_manager.cc:1397] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.007s
I20260806 04:06:45.871173 5105 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.873742 5123 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.874233 5123 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.876550 5092 raft_consensus.cc:348] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.877525 5092 raft_consensus.cc:374] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.877907 5092 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.879277 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.878958 5092 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.880172 4818 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:45.881629 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:45.883512 4992 leader_election.cc:341] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.884763 5123 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:45.892484 4996 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:45.892827 5097 ts_tablet_manager.cc:1428] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.049s user 0.013s sys 0.016s
I20260806 04:06:45.893643 5123 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.893906 5097 tablet_bootstrap.cc:492] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:45.894037 5123 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.894482 5123 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.899463 5092 ts_tablet_manager.cc:1428] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.027s user 0.007s sys 0.002s
I20260806 04:06:45.900719 5092 tablet_bootstrap.cc:492] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:45.904196 5123 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.904531 4669 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:45.906086 5097 tablet_bootstrap.cc:654] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.907501 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:45.908272 4818 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.914300 5093 tablet_bootstrap.cc:492] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:45.917371 4818 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:45.918800 4996 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:45.919754 5122 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:45.922165 5092 tablet_bootstrap.cc:654] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.923215 5093 ts_tablet_manager.cc:1397] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.077s user 0.031s sys 0.039s
I20260806 04:06:45.923092 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:06:45.925012 4992 leader_election.cc:341] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.925873 5123 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:06:45.928014 4882 consensus_peers.cc:487] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 -> Peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Couldn't send request to peer df2f1e7811114b8a995f6771fb6a6c8a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:45.928740 5093 raft_consensus.cc:348] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.929790 5093 raft_consensus.cc:374] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.931008 5093 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.932937 5122 raft_consensus.cc:686] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:45.932154 5093 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.934535 5097 tablet_bootstrap.cc:492] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:45.935384 5097 ts_tablet_manager.cc:1397] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.004s
I20260806 04:06:45.936731 5093 ts_tablet_manager.cc:1428] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20260806 04:06:45.938925 5093 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:45.940455 5097 raft_consensus.cc:348] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.941262 5097 raft_consensus.cc:374] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.944797 5092 tablet_bootstrap.cc:492] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:45.945492 5092 ts_tablet_manager.cc:1397] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.009s
I20260806 04:06:45.946112 5093 tablet_bootstrap.cc:654] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.941577 5097 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.947351 5097 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.949570 5097 ts_tablet_manager.cc:1428] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20260806 04:06:45.950450 5097 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:45.954344 5160 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.955013 5160 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.956750 5097 tablet_bootstrap.cc:654] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.956758 5122 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.960079 5092 raft_consensus.cc:348] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.960767 5092 raft_consensus.cc:374] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.961054 5092 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:45.961649 5092 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.964018 5092 ts_tablet_manager.cc:1428] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260806 04:06:45.964656 5103 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:45.964915 5092 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:45.965149 5103 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.965773 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:45.967320 5103 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:45.968241 4768 leader_election.cc:341] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.969326 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:45.971220 4876 leader_election.cc:341] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:45.972182 5092 tablet_bootstrap.cc:654] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:45.972096 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:45.972882 4818 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:45.975325 4882 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:45.975363 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:45.973589 5160 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.976303 5037 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:45.976413 5103 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.977694 5103 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.978168 5103 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.980376 5093 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:45.981137 5093 ts_tablet_manager.cc:1397] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.021s
I20260806 04:06:45.982093 4768 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:45.983603 5160 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:45.983953 5160 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:45.984288 5160 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.989512 5103 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:45.993033 5103 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:45.993119 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:45.994231 4818 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:45.995664 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:45.998391 4876 leader_election.cc:341] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.000140 5093 raft_consensus.cc:348] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.000948 5093 raft_consensus.cc:374] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:45.994012 4669 catalog_manager.cc:5495] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:46.001371 5093 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.003629 5160 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.003595 5093 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.006126 5160 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.006996 5092 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:46.007513 5092 ts_tablet_manager.cc:1397] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.001s
I20260806 04:06:46.007447 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:46.007993 5097 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.008530 5097 ts_tablet_manager.cc:1397] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.017s
W20260806 04:06:46.009308 4768 leader_election.cc:341] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.007501 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:46.010170 5037 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.010635 5092 raft_consensus.cc:348] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.010803 5097 raft_consensus.cc:348] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.011345 5092 raft_consensus.cc:374] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.011710 5092 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.012179 4818 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:46.011397 5097 raft_consensus.cc:374] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.012895 5093 ts_tablet_manager.cc:1428] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.031s user 0.005s sys 0.004s
I20260806 04:06:46.013171 5097 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.013607 4882 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.014257 5093 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.013864 5097 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.014941 5103 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.015443 5103 raft_consensus.cc:686] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:46.016240 5103 consensus_queue.cc:237] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.020005 5097 ts_tablet_manager.cc:1428] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260806 04:06:46.020926 5097 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.021311 5093 tablet_bootstrap.cc:654] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.025398 5037 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:46.026167 5097 tablet_bootstrap.cc:654] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.027482 4768 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:46.029083 5160 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.029680 5160 raft_consensus.cc:686] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:46.030769 5160 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.035035 5092 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.038199 5093 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.040009 5093 ts_tablet_manager.cc:1397] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.016s
I20260806 04:06:46.041564 5092 ts_tablet_manager.cc:1428] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260806 04:06:46.042785 5092 tablet_bootstrap.cc:492] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:46.042809 5093 raft_consensus.cc:348] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.044709 5097 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.044919 5093 raft_consensus.cc:374] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.045167 5097 ts_tablet_manager.cc:1397] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.007s
I20260806 04:06:46.045209 5093 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.046178 5093 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.047797 5097 raft_consensus.cc:348] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.048453 5093 ts_tablet_manager.cc:1428] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:06:46.049309 5093 tablet_bootstrap.cc:492] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.048475 5097 raft_consensus.cc:374] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.052762 5097 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.053028 5092 tablet_bootstrap.cc:654] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.054477 5097 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.055531 5105 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.055974 5105 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:06:46.058027 5105 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.057844 5103 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:46.058280 5093 tablet_bootstrap.cc:654] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.062923 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.065968 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.068063 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.068256 4674 catalog_manager.cc:5495] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:46.069635 5160 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:46.074901 4992 consensus_peers.cc:487] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 -> Peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Couldn't send request to peer 22fe557f852d4415a96fd683f970cc24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:46.075958 5160 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.076575 5160 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.079329 5160 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.077518 4672 catalog_manager.cc:5495] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:46.082448 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.097721 4930 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:46.083557 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.098487 5129 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:06:46.099975 4768 leader_election.cc:341] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.100670 4768 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.107558 5160 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:46.107959 5160 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:46.107918 5122 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.108489 5160 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.090556 5097 ts_tablet_manager.cc:1428] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260806 04:06:46.093034 5103 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.122232 5103 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.125160 5103 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.128671 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.131649 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.132871 5038 raft_consensus.cc:2379] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:46.135044 4876 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.136921 5129 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.140586 5097 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.150496 5129 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.159247 5105 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.160043 5105 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.163429 5160 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.165763 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.167517 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:46.167790 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.168327 4931 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.175694 4817 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.162395 5105 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.183544 5093 tablet_bootstrap.cc:492] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.178032 4768 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.184101 5093 ts_tablet_manager.cc:1397] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.135s user 0.015s sys 0.033s
I20260806 04:06:46.189703 5160 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.196704 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:46.198925 4768 leader_election.cc:341] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.198549 5093 raft_consensus.cc:348] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.203722 5093 raft_consensus.cc:374] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.204043 5093 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.203012 5105 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.204768 5093 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.207260 4931 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:46.207597 5097 tablet_bootstrap.cc:654] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.209347 5093 ts_tablet_manager.cc:1428] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260806 04:06:46.214915 5093 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.216703 5103 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 04:06:46.227460 4768 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.228652 5105 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.229179 5105 raft_consensus.cc:686] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:46.230086 5105 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.231769 5093 tablet_bootstrap.cc:654] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.245458 5092 tablet_bootstrap.cc:492] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:46.245973 5092 ts_tablet_manager.cc:1397] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.203s user 0.064s sys 0.067s
I20260806 04:06:46.248899 5092 raft_consensus.cc:348] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.249665 5092 raft_consensus.cc:374] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.250006 5092 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.259616 5097 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.260118 5097 ts_tablet_manager.cc:1397] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.120s user 0.022s sys 0.015s
I20260806 04:06:46.250700 5092 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.262259 5122 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.262988 5122 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.266755 5093 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.265075 5097 raft_consensus.cc:348] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.267241 5093 ts_tablet_manager.cc:1397] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.003s
I20260806 04:06:46.267362 5097 raft_consensus.cc:374] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.267697 5097 raft_consensus.cc:729] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.268373 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.268442 5097 consensus_queue.cc:260] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.269397 4818 raft_consensus.cc:2379] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:46.269717 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.270957 5097 ts_tablet_manager.cc:1428] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260806 04:06:46.270929 5122 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.271246 4992 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.271970 5097 tablet_bootstrap.cc:492] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.272527 5122 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.275319 5093 raft_consensus.cc:348] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.276084 5093 raft_consensus.cc:374] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.276432 5093 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.277550 5093 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.278268 5097 tablet_bootstrap.cc:654] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.278321 5092 ts_tablet_manager.cc:1428] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20260806 04:06:46.279336 5092 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:46.281714 5093 ts_tablet_manager.cc:1428] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260806 04:06:46.282716 5093 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.282804 5122 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.288506 4672 catalog_manager.cc:5495] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:46.289786 5122 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.290237 5122 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.292183 5122 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.292958 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.293316 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.294003 4931 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
W20260806 04:06:46.294435 4996 leader_election.cc:341] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.297631 5097 tablet_bootstrap.cc:492] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.297971 5097 ts_tablet_manager.cc:1397] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.002s
I20260806 04:06:46.299430 4992 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.299852 5092 tablet_bootstrap.cc:654] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.301199 5122 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:46.300873 5097 raft_consensus.cc:348] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.301594 5122 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:46.301752 5097 raft_consensus.cc:374] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.302023 5122 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.302150 5097 raft_consensus.cc:729] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.302950 5097 consensus_queue.cc:260] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.305409 5097 ts_tablet_manager.cc:1428] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:06:46.305683 5093 tablet_bootstrap.cc:654] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.306388 5097 tablet_bootstrap.cc:492] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.313828 5097 tablet_bootstrap.cc:654] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.315078 5122 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.318992 5103 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.319501 5103 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.321684 5103 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.323643 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.323751 5105 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.324513 5105 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.318409 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:06:46.326530 4876 leader_election.cc:341] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:46.328625 4996 leader_election.cc:341] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.328555 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.329854 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:46.330545 4768 leader_election.cc:341] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.330632 4817 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:46.331892 5105 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.332005 4882 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.333292 5092 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:46.334501 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.335036 5092 ts_tablet_manager.cc:1397] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.006s
I20260806 04:06:46.335451 5103 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:46.333288 5122 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:06:46.335862 4768 leader_election.cc:341] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.335850 5103 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:46.336742 5103 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.337689 5092 raft_consensus.cc:348] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.338224 5092 raft_consensus.cc:374] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.338434 5097 tablet_bootstrap.cc:492] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.336340 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:46.339602 5097 ts_tablet_manager.cc:1397] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.013s
I20260806 04:06:46.339736 4931 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.340054 5093 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.340462 5093 ts_tablet_manager.cc:1397] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.020s
I20260806 04:06:46.342509 5097 raft_consensus.cc:348] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.343209 5097 raft_consensus.cc:374] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.343619 5097 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.343832 5093 raft_consensus.cc:348] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.344657 5093 raft_consensus.cc:374] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.344413 5097 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.344535 5103 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.344959 5093 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.336341 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.345635 5093 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.345587 5122 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.346923 5122 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.348007 5093 ts_tablet_manager.cc:1428] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260806 04:06:46.349220 5093 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.349125 5122 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.350126 5105 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.351114 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.351768 4930 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:46.352029 5097 ts_tablet_manager.cc:1428] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20260806 04:06:46.352942 5097 tablet_bootstrap.cc:492] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.354275 5103 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.355356 5093 tablet_bootstrap.cc:654] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.355616 4992 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:46.357245 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:46.357435 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.358242 4817 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
W20260806 04:06:46.358708 4876 leader_election.cc:341] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.359143 5097 tablet_bootstrap.cc:654] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.359443 5122 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:46.359853 5122 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:46.360271 5122 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.360653 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:46.361491 4818 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.369065 5103 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.369558 5103 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.372882 5122 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.373564 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.375303 5122 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:06:46.377162 4876 leader_election.cc:341] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.376989 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:46.377856 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:46.377900 4817 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.378268 4930 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.378926 5122 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:46.382331 5097 tablet_bootstrap.cc:492] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.382820 5097 ts_tablet_manager.cc:1397] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.006s
I20260806 04:06:46.382341 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.384601 5105 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.385720 5097 raft_consensus.cc:348] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.386483 5097 raft_consensus.cc:374] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.386768 5097 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
W20260806 04:06:46.387483 4882 leader_election.cc:341] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.387586 5097 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.388020 4882 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.389722 5097 ts_tablet_manager.cc:1428] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:06:46.388841 4816 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.390618 5097 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.391376 5186 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.394932 5103 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.399394 4817 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:46.338465 5092 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.387104 5093 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.403626 5093 ts_tablet_manager.cc:1397] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.015s
I20260806 04:06:46.403869 5122 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.403782 5092 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.404541 5122 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.406725 5093 raft_consensus.cc:348] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.408733 4930 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:46.408869 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.410804 4927 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:46.411020 5122 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.412276 4992 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:46.412992 5093 raft_consensus.cc:374] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.413717 5093 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.414629 5093 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.415405 5092 ts_tablet_manager.cc:1428] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.080s user 0.006s sys 0.001s
I20260806 04:06:46.416611 5092 tablet_bootstrap.cc:492] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:46.412667 5105 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.412286 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.417979 5105 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.418093 4992 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:06:46.419584 4996 leader_election.cc:341] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.419685 5165 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.420179 5105 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.420408 5165 raft_consensus.cc:686] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:46.420554 5122 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:46.424211 5093 ts_tablet_manager.cc:1428] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.020s user 0.004s sys 0.003s
I20260806 04:06:46.424968 5093 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.423000 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.423133 5122 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:46.427098 5122 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.430186 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.430975 4818 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:46.421700 5165 consensus_queue.cc:237] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:46.431941 4768 leader_election.cc:341] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.435571 4882 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.442363 5105 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.443034 5105 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:46.449678 4992 consensus_peers.cc:487] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 -> Peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Couldn't send request to peer 22fe557f852d4415a96fd683f970cc24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:46.444585 5103 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.451313 5103 raft_consensus.cc:686] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:46.452602 5097 tablet_bootstrap.cc:654] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
W20260806 04:06:46.452804 4768 leader_election.cc:341] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.456315 4768 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.457719 5160 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.452419 5103 consensus_queue.cc:237] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.460671 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.457294 5105 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.468319 5105 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:06:46.471239 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:46.475524 5092 tablet_bootstrap.cc:654] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.454257 4931 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:46.482244 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.483191 4928 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:46.477818 5186 consensus_queue.cc:1035] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260806 04:06:46.486619 4992 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.487311 5207 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.487862 5207 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.488617 4674 catalog_manager.cc:5495] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:46.490459 5207 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.498085 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:46.496770 4768 leader_election.cc:341] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.499218 5122 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.499482 4768 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.501523 5105 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:46.502625 5105 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:46.503018 5105 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.510195 5122 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.498605 5207 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.513540 5207 raft_consensus.cc:686] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:46.514770 5207 consensus_queue.cc:237] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.519769 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:06:46.522874 4996 leader_election.cc:341] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.521495 5105 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.523612 4931 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.526809 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:46.527384 4928 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.545231 5093 tablet_bootstrap.cc:654] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.541992 5105 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:46.545485 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 59002.6us
I20260806 04:06:46.555384 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.556313 4930 raft_consensus.cc:2379] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:46.564450 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:46.565253 4992 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.566040 4930 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.572455 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:46.596158 4928 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:46.598114 4768 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:46.599416 5105 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.600087 5105 raft_consensus.cc:686] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:46.601286 5105 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.604753 5097 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
W20260806 04:06:46.605865 4768 leader_election.cc:341] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.607560 5092 tablet_bootstrap.cc:492] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:46.608028 5092 ts_tablet_manager.cc:1397] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.192s user 0.015s sys 0.000s
I20260806 04:06:46.608681 5191 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.614609 5092 raft_consensus.cc:348] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.615513 5092 raft_consensus.cc:374] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.615700 5213 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.615911 5092 raft_consensus.cc:729] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.616933 5092 consensus_queue.cc:260] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.619369 5092 ts_tablet_manager.cc:1428] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260806 04:06:46.620358 5092 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:46.627275 5097 ts_tablet_manager.cc:1397] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.237s user 0.014s sys 0.031s
I20260806 04:06:46.630139 5213 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.632581 5097 raft_consensus.cc:348] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.633381 5191 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.633994 5191 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.630074 5092 tablet_bootstrap.cc:654] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.636117 5093 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.636600 5093 ts_tablet_manager.cc:1397] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.212s user 0.009s sys 0.029s
I20260806 04:06:46.633175 5097 raft_consensus.cc:374] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.639456 5097 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.639492 5093 raft_consensus.cc:348] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.640352 5129 consensus_queue.cc:1035] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 04:06:46.645197 5093 raft_consensus.cc:374] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.645463 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.645488 5093 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.648123 5093 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.639418 4672 catalog_manager.cc:5495] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:46.645509 5097 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.651830 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.653211 4928 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:46.654691 4768 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.655190 5097 ts_tablet_manager.cc:1428] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.025s user 0.004s sys 0.004s
I20260806 04:06:46.655555 5160 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.656114 5097 tablet_bootstrap.cc:492] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:46.658589 5093 ts_tablet_manager.cc:1428] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.022s user 0.009s sys 0.001s
I20260806 04:06:46.659520 5093 tablet_bootstrap.cc:492] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.665254 5160 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.652077 5191 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.697358 5191 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.703069 5093 tablet_bootstrap.cc:654] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.766681 4674 catalog_manager.cc:5495] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:46.775882 5165 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.776530 5165 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.779122 5191 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.780902 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:46.781888 4814 raft_consensus.cc:2379] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:46.783965 5165 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:46.773138 4930 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:46.792586 5092 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:46.793134 5092 ts_tablet_manager.cc:1397] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.018s
I20260806 04:06:46.796888 5097 tablet_bootstrap.cc:654] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.798342 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.798753 5092 raft_consensus.cc:348] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.803794 5092 raft_consensus.cc:374] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.804533 5092 raft_consensus.cc:729] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.801756 4992 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.823372 5165 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:46.824179 5165 raft_consensus.cc:686] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:46.825363 5165 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.825208 4672 catalog_manager.cc:5495] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:46.831462 4992 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.861385 5213 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.871232 5213 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:46.885681 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 54342.9us
I20260806 04:06:46.887876 5233 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.892839 5092 consensus_queue.cc:260] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.902961 5160 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.903568 5160 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.905973 5160 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.908301 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:46.909437 4928 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:46.911212 5093 tablet_bootstrap.cc:492] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:46.911847 5093 ts_tablet_manager.cc:1397] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.252s user 0.025s sys 0.016s
I20260806 04:06:46.912592 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.914397 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:46.914510 5093 raft_consensus.cc:348] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.915755 5160 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:46.915771 5093 raft_consensus.cc:374] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.923323 5093 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.924132 5093 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.926791 5093 ts_tablet_manager.cc:1428] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260806 04:06:46.927889 5093 tablet_bootstrap.cc:492] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:46.902915 5092 ts_tablet_manager.cc:1428] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.109s user 0.008s sys 0.002s
I20260806 04:06:46.943367 5092 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:46.947362 5160 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.949404 5234 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.949993 5234 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.951395 5186 consensus_queue.cc:1035] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.957150 5093 tablet_bootstrap.cc:654] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.963361 5234 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:46.964082 5186 consensus_queue.cc:1035] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:46.965238 5097 tablet_bootstrap.cc:492] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:46.965520 5092 tablet_bootstrap.cc:654] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:46.965750 5097 ts_tablet_manager.cc:1397] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.310s user 0.035s sys 0.039s
I20260806 04:06:46.966856 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:46.968441 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:46.969288 4876 leader_election.cc:341] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:46.977097 4882 leader_election.cc:341] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:46.980564 4882 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:46.981618 5186 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:46.988489 5097 raft_consensus.cc:348] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.989398 5097 raft_consensus.cc:374] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:46.989719 5097 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:46.991180 5097 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.993616 4674 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:46.997500 5191 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.998732 5097 ts_tablet_manager.cc:1428] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20260806 04:06:46.998613 5186 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:46.998930 5191 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:46.999253 5186 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.001124 5186 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.001253 5191 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.001937 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.002723 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.004029 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:47.006704 4768 leader_election.cc:341] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.006624 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.008361 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.012138 5093 tablet_bootstrap.cc:492] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.012653 5093 ts_tablet_manager.cc:1397] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.005s
W20260806 04:06:47.013944 4768 leader_election.cc:341] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.015718 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.012880 5186 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.019840 5097 tablet_bootstrap.cc:492] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.016899 5191 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.026223 5093 raft_consensus.cc:348] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.026989 5093 raft_consensus.cc:374] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.027271 5093 raft_consensus.cc:729] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.033649 5097 tablet_bootstrap.cc:654] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.033282 5165 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:47.050138 5093 consensus_queue.cc:260] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.053541 5093 ts_tablet_manager.cc:1428] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.041s user 0.007s sys 0.003s
I20260806 04:06:47.055086 5093 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.056619 5097 tablet_bootstrap.cc:492] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.057126 5097 ts_tablet_manager.cc:1397] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.007s
I20260806 04:06:47.065361 5165 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:47.066316 5092 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:47.067059 5092 ts_tablet_manager.cc:1397] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.124s user 0.021s sys 0.006s
I20260806 04:06:47.070322 5097 raft_consensus.cc:348] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.071096 5097 raft_consensus.cc:374] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.071377 5097 raft_consensus.cc:729] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.072038 5097 consensus_queue.cc:260] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.074175 5097 ts_tablet_manager.cc:1428] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20260806 04:06:47.067287 5186 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.074899 5186 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.076740 5097 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.082442 5092 raft_consensus.cc:348] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.083142 5092 raft_consensus.cc:374] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.083379 5092 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.087064 5092 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.087936 5093 tablet_bootstrap.cc:654] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.088593 4928 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.092204 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.093207 5186 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.093986 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.095861 4876 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.096799 5186 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.097067 5092 ts_tablet_manager.cc:1428] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260806 04:06:47.100756 5092 tablet_bootstrap.cc:492] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.102721 5207 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:47.105234 5097 tablet_bootstrap.cc:654] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.107035 5092 tablet_bootstrap.cc:654] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.118918 5093 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.119421 5093 ts_tablet_manager.cc:1397] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.002s
I20260806 04:06:47.120229 4930 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.119426 5213 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:47.125663 5097 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.126324 5097 ts_tablet_manager.cc:1397] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.016s
I20260806 04:06:47.135501 5093 raft_consensus.cc:348] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.136572 5093 raft_consensus.cc:374] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.136915 5093 raft_consensus.cc:729] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.137938 5097 raft_consensus.cc:348] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.139197 5097 raft_consensus.cc:374] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.139557 5097 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.140386 5093 consensus_queue.cc:260] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.142750 5097 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.144465 5093 ts_tablet_manager.cc:1428] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20260806 04:06:47.145462 5097 ts_tablet_manager.cc:1428] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20260806 04:06:47.147210 5097 tablet_bootstrap.cc:492] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.148540 5093 tablet_bootstrap.cc:492] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.155439 5092 tablet_bootstrap.cc:492] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:47.156339 5092 ts_tablet_manager.cc:1397] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.003s
W20260806 04:06:47.158042 4768 consensus_peers.cc:487] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:47.179788 5191 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:47.172899 5092 raft_consensus.cc:348] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.189124 5092 raft_consensus.cc:374] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.189623 5092 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.193168 5186 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.193987 5186 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.201793 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.201988 5186 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.202725 5038 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:47.190584 5092 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.204421 4876 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:47.203341 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.210887 5092 ts_tablet_manager.cc:1428] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.054s user 0.008s sys 0.002s
I20260806 04:06:47.212054 5186 raft_consensus.cc:2781] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.212605 5186 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.213037 5186 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.214777 5092 tablet_bootstrap.cc:492] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.216959 5093 tablet_bootstrap.cc:654] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.219492 5097 tablet_bootstrap.cc:654] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.229286 5186 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.218588 4814 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:47.232983 5186 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.233196 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:47.233498 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:47.233861 5038 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.234045 4814 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.257411 5038 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:47.259254 4876 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:47.260422 5186 raft_consensus.cc:2781] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:47.261126 5186 raft_consensus.cc:686] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:47.262246 5186 consensus_queue.cc:237] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.272095 5092 tablet_bootstrap.cc:654] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.277788 4814 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:47.279136 5093 tablet_bootstrap.cc:492] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.279743 5093 ts_tablet_manager.cc:1397] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.133s user 0.012s sys 0.003s
I20260806 04:06:47.282826 5092 tablet_bootstrap.cc:492] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:47.283447 5092 ts_tablet_manager.cc:1397] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.001s
I20260806 04:06:47.282910 5093 raft_consensus.cc:348] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.284076 5093 raft_consensus.cc:374] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.284472 5093 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.285287 5093 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.286530 5097 tablet_bootstrap.cc:492] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.287029 5097 ts_tablet_manager.cc:1397] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.140s user 0.042s sys 0.007s
I20260806 04:06:47.288446 5093 ts_tablet_manager.cc:1428] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260806 04:06:47.288378 5092 raft_consensus.cc:348] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.289254 5092 raft_consensus.cc:374] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.289558 5092 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.289012 4672 catalog_manager.cc:5495] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:47.289700 5097 raft_consensus.cc:348] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.290182 5093 tablet_bootstrap.cc:492] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.290372 5097 raft_consensus.cc:374] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.290881 5097 raft_consensus.cc:729] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.290354 5092 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.291517 5097 consensus_queue.cc:260] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.293572 5097 ts_tablet_manager.cc:1428] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:06:47.298205 5092 ts_tablet_manager.cc:1428] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260806 04:06:47.299314 5092 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.306118 5092 tablet_bootstrap.cc:654] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.308338 5097 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.308810 5263 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:47.317380 5093 tablet_bootstrap.cc:654] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.323266 5097 tablet_bootstrap.cc:654] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.334967 5093 tablet_bootstrap.cc:492] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.335592 5093 ts_tablet_manager.cc:1397] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.002s
I20260806 04:06:47.339735 5093 raft_consensus.cc:348] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.340695 5093 raft_consensus.cc:374] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.341285 5093 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.342445 5093 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.345221 5093 ts_tablet_manager.cc:1428] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 04:06:47.346498 5093 tablet_bootstrap.cc:492] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.368089 5093 tablet_bootstrap.cc:654] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.407732 5093 tablet_bootstrap.cc:492] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.408321 5093 ts_tablet_manager.cc:1397] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.062s user 0.033s sys 0.022s
I20260806 04:06:47.410027 5092 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:47.410514 5092 ts_tablet_manager.cc:1397] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.111s user 0.030s sys 0.039s
I20260806 04:06:47.411017 5165 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:47.412894 5160 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.413424 5160 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.415820 5160 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.417786 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.418303 5093 raft_consensus.cc:348] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.419080 5093 raft_consensus.cc:374] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:06:47.419301 4768 leader_election.cc:341] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.419349 5093 raft_consensus.cc:729] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.421348 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.421963 5093 consensus_queue.cc:260] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:47.423991 4768 leader_election.cc:341] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.424042 5097 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.426697 5097 ts_tablet_manager.cc:1397] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.119s user 0.038s sys 0.029s
I20260806 04:06:47.425959 5093 ts_tablet_manager.cc:1428] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260806 04:06:47.425268 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.430372 5097 raft_consensus.cc:348] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.431353 5097 raft_consensus.cc:374] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.431715 5097 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.432243 5160 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.425266 5092 raft_consensus.cc:348] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.432963 5092 raft_consensus.cc:374] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.432407 5097 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.433257 5092 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.433676 5093 tablet_bootstrap.cc:492] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.434015 5092 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.439548 5097 ts_tablet_manager.cc:1428] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260806 04:06:47.440503 5097 tablet_bootstrap.cc:492] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.440956 5093 tablet_bootstrap.cc:654] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.440938 5092 ts_tablet_manager.cc:1428] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260806 04:06:47.446627 5092 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.448700 5097 tablet_bootstrap.cc:654] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.458685 5186 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.459573 5186 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.462950 5092 tablet_bootstrap.cc:654] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.461964 5186 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.464030 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.464740 5038 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:47.466336 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.467058 4814 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:47.468261 4882 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.469103 5234 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.469444 5234 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.469750 5234 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.480697 5234 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.483798 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:47.484452 4814 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.491932 4814 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:47.493006 4882 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.493851 5186 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:47.494340 5186 raft_consensus.cc:686] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:47.495190 5186 consensus_queue.cc:237] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.500236 5092 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
W20260806 04:06:47.500306 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:47.500981 5092 ts_tablet_manager.cc:1397] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.021s
I20260806 04:06:47.501207 5234 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.503049 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:47.503835 5093 tablet_bootstrap.cc:492] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.503861 5038 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.503823 5092 raft_consensus.cc:348] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.504451 5093 ts_tablet_manager.cc:1397] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.071s user 0.020s sys 0.012s
I20260806 04:06:47.504663 5092 raft_consensus.cc:374] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.504942 5092 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.505681 5092 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.506873 5093 raft_consensus.cc:348] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.507512 5093 raft_consensus.cc:374] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.507735 5093 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.508302 5093 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.511358 5038 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:47.512005 5097 tablet_bootstrap.cc:492] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.512435 5097 ts_tablet_manager.cc:1397] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.010s
I20260806 04:06:47.514827 5097 raft_consensus.cc:348] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.515252 5092 ts_tablet_manager.cc:1428] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260806 04:06:47.515506 5097 raft_consensus.cc:374] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.515775 5097 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.516189 5093 ts_tablet_manager.cc:1428] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20260806 04:06:47.516444 5097 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.518220 5093 tablet_bootstrap.cc:492] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.518509 5097 ts_tablet_manager.cc:1428] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:06:47.520920 5092 tablet_bootstrap.cc:492] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.523295 4669 catalog_manager.cc:5495] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:47.524422 5093 tablet_bootstrap.cc:654] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.527933 5097 tablet_bootstrap.cc:492] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.531280 5186 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.531826 5186 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.534092 5105 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.534559 5105 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.535930 5097 tablet_bootstrap.cc:654] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.536520 5105 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.538385 5092 tablet_bootstrap.cc:654] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.539868 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.540537 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.541222 4931 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
W20260806 04:06:47.543263 4768 leader_election.cc:341] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.544616 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.545894 5105 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.546355 5105 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.546824 5105 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.547102 5093 tablet_bootstrap.cc:492] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.547720 5093 ts_tablet_manager.cc:1397] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.016s
I20260806 04:06:47.550275 5093 raft_consensus.cc:348] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.551069 5093 raft_consensus.cc:374] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.551396 5093 raft_consensus.cc:729] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.552124 5093 consensus_queue.cc:260] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.555843 5093 ts_tablet_manager.cc:1428] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 04:06:47.557158 5093 tablet_bootstrap.cc:492] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.567988 5093 tablet_bootstrap.cc:654] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.567909 5038 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.571509 5160 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.572022 5160 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.574638 5160 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.576402 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:47.578622 4768 leader_election.cc:341] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.579238 5207 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.579429 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.580019 5207 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:47.582901 4768 leader_election.cc:341] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.583742 4768 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.585080 5160 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.586823 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.587842 4814 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:47.590301 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.590689 4996 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.591591 5213 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.592093 5213 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.592649 5213 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:47.595615 4992 leader_election.cc:341] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.605757 5160 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.586889 5207 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.606355 5160 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.609334 5160 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.612586 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:47.615420 4768 leader_election.cc:341] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.616079 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.618247 5207 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.619041 5207 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:47.620771 4768 leader_election.cc:341] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.622146 4768 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.622361 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.623544 5160 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:47.624162 4882 leader_election.cc:341] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.628157 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.629058 4814 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:47.631235 4996 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.632602 5165 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.633206 5165 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.632998 5105 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.633718 5165 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.637444 5213 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.641155 5093 tablet_bootstrap.cc:492] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.641918 5093 ts_tablet_manager.cc:1397] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.085s user 0.014s sys 0.008s
I20260806 04:06:47.649274 5093 raft_consensus.cc:348] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.650305 5093 raft_consensus.cc:374] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.656308 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.651310 5186 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:47.658249 4876 leader_election.cc:341] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.658932 4876 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.660240 5186 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.660337 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:47.661465 4814 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.664778 5105 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.667264 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:47.667780 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:47.668187 4931 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:47.670207 4768 leader_election.cc:341] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.670531 5186 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.671087 5186 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.671711 5207 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.650645 5093 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.673386 5093 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.676371 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.683395 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.683816 4930 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:47.677017 5186 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.679252 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.686035 5093 ts_tablet_manager.cc:1428] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.044s user 0.008s sys 0.000s
W20260806 04:06:47.687377 4876 leader_election.cc:341] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.681869 4931 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
W20260806 04:06:47.687983 4882 leader_election.cc:341] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.683094 4814 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:47.688516 4882 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.689360 5234 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.689860 5093 tablet_bootstrap.cc:492] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.690086 4996 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.691002 5207 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:47.692106 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.700731 5105 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:47.701936 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:47.703398 5105 raft_consensus.cc:686] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:06:47.703958 4992 leader_election.cc:341] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.704797 5105 consensus_queue.cc:237] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.700896 5213 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.728302 5097 tablet_bootstrap.cc:492] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.728940 5097 ts_tablet_manager.cc:1397] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.202s user 0.048s sys 0.088s
I20260806 04:06:47.734157 5092 tablet_bootstrap.cc:492] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:47.739215 5097 raft_consensus.cc:348] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.740300 5097 raft_consensus.cc:374] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.740638 5097 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.741420 5097 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.710603 5165 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.748898 5165 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.749293 5234 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:47.758056 5097 ts_tablet_manager.cc:1428] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s
I20260806 04:06:47.759459 5097 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.760792 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:47.761617 4931 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.766669 5092 ts_tablet_manager.cc:1397] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.246s user 0.059s sys 0.071s
W20260806 04:06:47.766414 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 53999.6us
I20260806 04:06:47.769503 5092 raft_consensus.cc:348] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.770404 5092 raft_consensus.cc:374] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.780442 5092 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.771304 5093 tablet_bootstrap.cc:654] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.781270 4672 catalog_manager.cc:5495] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:47.771508 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:47.783027 4816 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.783886 5092 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.785135 5207 raft_consensus.cc:686] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:47.786476 5207 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.789547 5263 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:06:47.790270 5263 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.791349 5097 tablet_bootstrap.cc:654] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.795120 5092 ts_tablet_manager.cc:1428] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260806 04:06:47.796144 5092 tablet_bootstrap.cc:492] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.800969 5263 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.803957 4816 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:47.779834 4931 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:47.805855 5186 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.810808 4996 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.811820 5207 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:47.811483 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.812587 5207 raft_consensus.cc:686] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:47.813552 5093 tablet_bootstrap.cc:492] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.814095 5093 ts_tablet_manager.cc:1397] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.011s
I20260806 04:06:47.813603 5207 consensus_queue.cc:237] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.808141 5186 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.817109 5234 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:47.817989 5186 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.820363 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.821208 5092 tablet_bootstrap.cc:654] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.822141 5093 raft_consensus.cc:348] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.822986 5093 raft_consensus.cc:374] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.823266 5093 raft_consensus.cc:729] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.824862 4672 catalog_manager.cc:5495] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:47.827240 4992 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.827268 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.829701 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:47.829980 4882 leader_election.cc:341] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.831605 5295 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:06:47.832687 4876 leader_election.cc:341] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.833108 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.836088 5129 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.839274 5093 consensus_queue.cc:260] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.842110 5093 ts_tablet_manager.cc:1428] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.028s user 0.006s sys 0.007s
I20260806 04:06:47.844210 5093 tablet_bootstrap.cc:492] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.845717 5097 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.846328 5097 ts_tablet_manager.cc:1397] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.087s user 0.018s sys 0.010s
I20260806 04:06:47.849730 5093 tablet_bootstrap.cc:654] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.849794 5097 raft_consensus.cc:348] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.850884 5097 raft_consensus.cc:374] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.851269 5097 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.852205 5097 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.855247 5097 ts_tablet_manager.cc:1428] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260806 04:06:47.857053 5097 tablet_bootstrap.cc:492] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.864658 5097 tablet_bootstrap.cc:654] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.868089 5263 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.868279 5093 tablet_bootstrap.cc:492] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.868862 5093 ts_tablet_manager.cc:1397] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.006s
I20260806 04:06:47.868762 5263 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.871402 5093 raft_consensus.cc:348] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.871778 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.872167 5093 raft_consensus.cc:374] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.872437 5093 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.872390 4816 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:47.873225 5093 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.874159 4996 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.875638 5297 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.876012 5297 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.876456 5297 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.877064 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.877677 4931 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:47.880986 5093 ts_tablet_manager.cc:1428] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260806 04:06:47.881496 5263 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.883894 5097 tablet_bootstrap.cc:492] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:47.884980 5097 ts_tablet_manager.cc:1397] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.003s
I20260806 04:06:47.888545 5097 raft_consensus.cc:348] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.889569 5097 raft_consensus.cc:374] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.889901 5097 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.891826 5093 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:47.895217 5297 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.897117 5097 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.900110 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:47.900784 4931 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.901830 5097 ts_tablet_manager.cc:1428] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260806 04:06:47.902788 5097 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:47.899521 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:47.906152 4816 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.909077 4931 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:47.910223 4992 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:47.915160 5092 tablet_bootstrap.cc:492] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:47.915635 5092 ts_tablet_manager.cc:1397] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.120s user 0.019s sys 0.010s
I20260806 04:06:47.916055 4669 catalog_manager.cc:5495] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:47.899031 5297 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.916687 5207 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:47.916541 4816 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:47.926618 5092 raft_consensus.cc:348] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.927891 5092 raft_consensus.cc:374] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:47.928258 5092 raft_consensus.cc:729] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:47.928972 5092 consensus_queue.cc:260] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.926841 5105 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.927659 5207 raft_consensus.cc:686] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:47.931396 5105 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.932279 5207 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.936170 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.937767 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:47.937984 4768 leader_election.cc:341] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:47.940182 4768 leader_election.cc:341] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.940660 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:47.941612 5160 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:47.943255 5093 tablet_bootstrap.cc:654] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.944557 5186 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.945544 5186 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.947139 5092 ts_tablet_manager.cc:1428] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.031s user 0.004s sys 0.003s
I20260806 04:06:47.948207 5092 tablet_bootstrap.cc:492] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:47.948792 5186 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.950860 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.938370 5105 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.952368 4814 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:47.952958 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.954144 5097 tablet_bootstrap.cc:654] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.954373 4882 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:47.955348 5297 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.955916 5234 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:47.956373 5234 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:47.956808 5234 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:47.957202 4876 leader_election.cc:341] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.956280 5297 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.962494 5297 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:47.963640 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:47.964608 4931 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:47.968080 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:47.970501 4996 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:47.970507 4669 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:47.972044 5207 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.968027 5234 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.978539 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:47.979457 4816 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:47.980225 5092 tablet_bootstrap.cc:654] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:47.983008 5129 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.983594 5129 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.984263 5234 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.985863 5129 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:47.987651 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:47.990100 5207 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:47.991163 4876 leader_election.cc:341] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.978535 5186 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:47.993172 5186 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:47.994779 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:47.995761 5093 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:47.996335 5093 ts_tablet_manager.cc:1397] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.114s user 0.026s sys 0.009s
W20260806 04:06:47.997890 4876 leader_election.cc:341] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:47.999912 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.000082 5093 raft_consensus.cc:348] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.000857 5093 raft_consensus.cc:374] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.001138 5093 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.002190 5097 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.001780 5093 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:48.003002 4882 leader_election.cc:341] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.003618 4882 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:48.004772 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.005575 5092 tablet_bootstrap.cc:492] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.005532 4818 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:48.006125 5092 ts_tablet_manager.cc:1397] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.003s
I20260806 04:06:48.003257 5097 ts_tablet_manager.cc:1397] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.101s user 0.014s sys 0.001s
I20260806 04:06:48.009169 5234 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.010260 5097 raft_consensus.cc:348] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.010988 5097 raft_consensus.cc:374] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.011271 5097 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.011859 5097 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.012393 4882 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.013152 5129 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.013477 5129 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.013800 5129 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.013913 5097 ts_tablet_manager.cc:1428] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 04:06:48.004267 5093 ts_tablet_manager.cc:1428] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:06:48.015097 5093 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:48.015228 5097 tablet_bootstrap.cc:492] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.020790 5092 raft_consensus.cc:348] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.021502 5092 raft_consensus.cc:374] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.021824 5092 raft_consensus.cc:729] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.022578 5092 consensus_queue.cc:260] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.024037 5097 tablet_bootstrap.cc:654] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.024785 5092 ts_tablet_manager.cc:1428] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260806 04:06:48.025640 5093 tablet_bootstrap.cc:654] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.029623 5092 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:48.031366 5186 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.033593 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.034318 5037 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:48.042794 5092 tablet_bootstrap.cc:654] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.042192 5186 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:48.052035 5093 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:48.052558 5093 ts_tablet_manager.cc:1397] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.009s
I20260806 04:06:48.053588 5097 tablet_bootstrap.cc:492] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.054176 5097 ts_tablet_manager.cc:1397] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.009s
I20260806 04:06:48.054927 5093 raft_consensus.cc:348] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.055703 5093 raft_consensus.cc:374] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.055986 5093 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.056622 5093 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.057364 5097 raft_consensus.cc:348] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.058126 5097 raft_consensus.cc:374] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.058398 5097 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.058905 5093 ts_tablet_manager.cc:1428] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260806 04:06:48.059005 5097 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.061136 5097 ts_tablet_manager.cc:1428] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260806 04:06:48.061476 5092 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.061980 5097 tablet_bootstrap.cc:492] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.061978 5092 ts_tablet_manager.cc:1397] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.002s
I20260806 04:06:48.062091 4816 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:48.063704 4882 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.064491 5129 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.065740 5186 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:48.068077 5093 tablet_bootstrap.cc:492] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:48.068265 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:48.068924 5037 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.069739 5097 tablet_bootstrap.cc:654] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.071722 5129 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.076586 5093 tablet_bootstrap.cc:654] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.094830 5234 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.066113 5092 raft_consensus.cc:348] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.102006 5092 raft_consensus.cc:374] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.115844 5234 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.116012 5092 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.116616 5234 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.103370 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:48.118975 4816 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.121650 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.122884 4814 raft_consensus.cc:2379] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:48.122957 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.098599 5037 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:48.125326 4876 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:48.117069 5092 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.111095 5105 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.128489 5105 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.131297 5105 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.133925 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:48.135919 4768 leader_election.cc:341] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.137660 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.138571 4931 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.142015 4768 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.147325 5129 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.149154 5092 ts_tablet_manager.cc:1428] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.086s user 0.007s sys 0.002s
I20260806 04:06:48.149256 5160 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.150008 5160 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.151414 5092 tablet_bootstrap.cc:492] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:48.126776 5234 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.107319 5186 raft_consensus.cc:686] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:48.154479 5097 tablet_bootstrap.cc:492] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.156847 5160 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.164697 5093 tablet_bootstrap.cc:492] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:48.164232 5186 consensus_queue.cc:237] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.165210 5093 ts_tablet_manager.cc:1397] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.097s user 0.018s sys 0.042s
I20260806 04:06:48.158123 5129 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.166111 5097 ts_tablet_manager.cc:1397] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.104s user 0.027s sys 0.064s
I20260806 04:06:48.169579 4816 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:48.171025 4882 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.176100 5093 raft_consensus.cc:348] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.177433 5093 raft_consensus.cc:374] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.177711 5093 raft_consensus.cc:729] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.179271 5093 consensus_queue.cc:260] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.171957 5160 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.175524 5129 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:48.185910 5093 ts_tablet_manager.cc:1428] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.020s user 0.005s sys 0.005s
I20260806 04:06:48.187273 5093 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:48.195398 5160 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.193439 5129 raft_consensus.cc:686] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:48.206804 5092 tablet_bootstrap.cc:654] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.207373 5097 raft_consensus.cc:348] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.208818 5097 raft_consensus.cc:374] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.209092 5097 raft_consensus.cc:729] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.209057 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:48.209430 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:48.210155 4931 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.209895 5097 consensus_queue.cc:260] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:48.210641 4768 leader_election.cc:341] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.213693 5097 ts_tablet_manager.cc:1428] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20260806 04:06:48.214589 5097 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.225713 5207 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:48.227950 5097 tablet_bootstrap.cc:654] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.207190 5160 consensus_queue.cc:1035] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260806 04:06:48.234936 5129 consensus_queue.cc:237] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.238365 4669 catalog_manager.cc:5495] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.239368 5093 tablet_bootstrap.cc:654] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.256685 4931 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:48.263025 5092 tablet_bootstrap.cc:492] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.263655 5092 ts_tablet_manager.cc:1397] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.113s user 0.014s sys 0.011s
I20260806 04:06:48.272989 5213 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:48.271989 5092 raft_consensus.cc:348] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.275065 5092 raft_consensus.cc:374] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.276154 5092 raft_consensus.cc:729] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.279331 4768 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.284559 5336 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:48.280043 5160 consensus_queue.cc:1035] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.285267 5336 raft_consensus.cc:686] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:48.289888 5097 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.290377 5097 ts_tablet_manager.cc:1397] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.076s user 0.018s sys 0.006s
I20260806 04:06:48.293318 5092 consensus_queue.cc:260] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.300557 5097 raft_consensus.cc:348] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.301326 5097 raft_consensus.cc:374] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.301581 5097 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.302759 5097 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.310840 5093 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:48.286393 5336 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.311373 5093 ts_tablet_manager.cc:1397] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.124s user 0.015s sys 0.001s
I20260806 04:06:48.313535 5160 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.314072 5160 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.315465 5093 raft_consensus.cc:348] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.316277 5093 raft_consensus.cc:374] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.316591 5093 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.317281 5093 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.317252 4669 catalog_manager.cc:5495] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.321357 5160 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.323770 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.324609 4930 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.327072 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:48.328822 4768 leader_election.cc:341] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.330199 5093 ts_tablet_manager.cc:1428] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260806 04:06:48.331234 5093 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:48.335889 5092 ts_tablet_manager.cc:1428] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.072s user 0.004s sys 0.003s
I20260806 04:06:48.337611 5097 ts_tablet_manager.cc:1428] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.047s user 0.010s sys 0.000s
I20260806 04:06:48.336874 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.339704 5160 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.340101 5160 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.340432 5160 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.344892 5097 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.352577 5160 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.356443 5093 tablet_bootstrap.cc:654] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.368106 4669 catalog_manager.cc:5495] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.371287 5336 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.371829 5336 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.374111 5336 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.375376 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.376070 4930 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.376216 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.377224 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.377596 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:48.378175 4768 leader_election.cc:341] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:48.379204 4768 leader_election.cc:341] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.379379 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:48.379814 5336 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.379703 5160 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.380293 5336 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.380990 5336 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.380225 4928 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.379312 5097 tablet_bootstrap.cc:654] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.384466 5160 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.385201 5160 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.387621 5160 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.389070 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.389745 4930 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.390758 5093 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:48.391397 5093 ts_tablet_manager.cc:1397] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.060s user 0.019s sys 0.004s
I20260806 04:06:48.391343 4768 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.392558 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.393314 5035 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.394788 5160 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.395287 5160 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.395694 5160 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.397635 5336 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.401412 4928 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:48.402679 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.403597 5334 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:48.404147 5334 raft_consensus.cc:686] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:48.404982 5186 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.405068 5334 consensus_queue.cc:237] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.405601 5186 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.407188 5097 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.407646 5097 ts_tablet_manager.cc:1397] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.007s
I20260806 04:06:48.407959 5186 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.408595 5160 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.362783 5092 tablet_bootstrap.cc:492] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:48.411332 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.411542 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:48.411808 5093 raft_consensus.cc:348] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.412463 5093 raft_consensus.cc:374] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.412237 4928 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.416468 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:48.417114 5035 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.417529 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:48.418126 4930 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.412829 5093 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.419353 5093 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.421773 5093 ts_tablet_manager.cc:1428] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260806 04:06:48.422847 5092 tablet_bootstrap.cc:654] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.423427 5093 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:48.425772 4930 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:48.427407 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.428467 5105 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:48.429019 5105 raft_consensus.cc:686] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:48.430373 5093 tablet_bootstrap.cc:654] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.429891 5105 consensus_queue.cc:237] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.430765 5336 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.435382 4928 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:48.435577 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:48.437407 4768 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.438359 5334 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
W20260806 04:06:48.438767 4768 leader_election.cc:341] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.438970 5334 raft_consensus.cc:686] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:48.439990 5334 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.413161 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:48.413602 4876 leader_election.cc:341] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.441615 5035 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:48.413812 5097 raft_consensus.cc:348] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.442569 5097 raft_consensus.cc:374] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.442888 5097 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.443567 5097 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.444356 5207 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:48.445763 5097 ts_tablet_manager.cc:1428] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.036s user 0.002s sys 0.002s
I20260806 04:06:48.414868 5160 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:48.448098 4882 leader_election.cc:341] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.448544 4882 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:48.449628 5129 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.451289 5097 tablet_bootstrap.cc:492] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.450644 4672 catalog_manager.cc:5495] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.452562 5297 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:48.451090 5263 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.458443 5263 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.462174 5263 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:48.464056 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.468127 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.469316 4818 raft_consensus.cc:2379] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:48.470611 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:48.471892 5263 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.480396 5092 tablet_bootstrap.cc:492] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.480907 5297 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:48.484454 5097 tablet_bootstrap.cc:654] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.487161 5092 ts_tablet_manager.cc:1397] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.011s
I20260806 04:06:48.490759 5263 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.495980 5092 raft_consensus.cc:348] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.496582 5092 raft_consensus.cc:374] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.496809 5092 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.497530 5092 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.502875 5092 ts_tablet_manager.cc:1428] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 04:06:48.503619 5092 tablet_bootstrap.cc:492] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:48.511759 4669 catalog_manager.cc:5495] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.513875 4669 catalog_manager.cc:5495] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.530460 5093 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:48.531028 5093 ts_tablet_manager.cc:1397] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.108s user 0.016s sys 0.017s
I20260806 04:06:48.534751 5207 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.538905 5093 raft_consensus.cc:348] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.539703 5093 raft_consensus.cc:374] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.540050 5093 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.551286 5093 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.557267 5105 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.559258 5092 tablet_bootstrap.cc:654] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.558768 5105 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.567216 5105 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.568254 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.568257 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:48.571547 4768 leader_election.cc:341] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.571923 5093 ts_tablet_manager.cc:1428] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260806 04:06:48.573419 5093 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
W20260806 04:06:48.590641 4768 leader_election.cc:341] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.591326 4768 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.592427 5105 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.597034 5092 tablet_bootstrap.cc:492] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.597668 5092 ts_tablet_manager.cc:1397] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.004s
I20260806 04:06:48.601719 5092 raft_consensus.cc:348] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.602581 5092 raft_consensus.cc:374] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.602993 5092 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.603803 5097 tablet_bootstrap.cc:492] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.604456 5097 ts_tablet_manager.cc:1397] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.158s user 0.012s sys 0.004s
I20260806 04:06:48.603809 5092 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.607172 5093 tablet_bootstrap.cc:654] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.608072 5092 ts_tablet_manager.cc:1428] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260806 04:06:48.607242 5097 raft_consensus.cc:348] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.609308 5097 raft_consensus.cc:374] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.609575 5097 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.612977 5097 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.618372 5092 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:48.620416 5097 ts_tablet_manager.cc:1428] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
W20260806 04:06:48.621236 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:48.625607 5097 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.626628 5207 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.627511 5207 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.629880 5092 tablet_bootstrap.cc:654] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.632552 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.636267 5207 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:48.637825 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.639137 4928 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:48.640431 4992 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:48.642213 5207 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.658556 5092 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.659296 5092 ts_tablet_manager.cc:1397] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.009s
I20260806 04:06:48.660702 5097 tablet_bootstrap.cc:654] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.661331 5093 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:48.666083 5093 ts_tablet_manager.cc:1397] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.093s user 0.020s sys 0.019s
I20260806 04:06:48.664827 5092 raft_consensus.cc:348] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.666853 5092 raft_consensus.cc:374] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.667140 5092 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.667827 5092 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.668612 5160 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.669427 5160 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.670364 5092 ts_tablet_manager.cc:1428] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260806 04:06:48.672183 5092 tablet_bootstrap.cc:492] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:48.672717 5093 raft_consensus.cc:348] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.674069 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.674368 5093 raft_consensus.cc:374] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.675176 5093 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.676959 5093 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.678339 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.683853 5037 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.675247 4928 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:48.687531 4768 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.688249 5207 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.689484 5105 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.691131 5105 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.691773 5105 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.686393 5160 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.695748 5337 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.696544 5337 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.698377 5093 ts_tablet_manager.cc:1428] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.032s user 0.011s sys 0.000s
I20260806 04:06:48.699806 5093 tablet_bootstrap.cc:492] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:48.716383 5337 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.740339 5097 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.740913 5097 ts_tablet_manager.cc:1397] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.116s user 0.010s sys 0.031s
I20260806 04:06:48.720880 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:48.743413 4876 leader_election.cc:341] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.741022 5129 consensus_queue.cc:1035] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 04:06:48.720976 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.752768 5097 raft_consensus.cc:348] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.753640 5097 raft_consensus.cc:374] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.753976 5097 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
W20260806 04:06:48.753515 4882 leader_election.cc:341] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.754895 4882 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:48.755144 5105 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.756773 5129 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:48.754879 5097 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.757890 5092 tablet_bootstrap.cc:654] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.759703 5097 ts_tablet_manager.cc:1428] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.018s user 0.007s sys 0.002s
I20260806 04:06:48.760848 5097 tablet_bootstrap.cc:492] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:48.765617 5105 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.766592 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:48.767972 4930 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.769280 5129 consensus_queue.cc:1035] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.772295 5234 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.772972 5234 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.775870 5234 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.780850 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:48.781771 4768 consensus_peers.cc:487] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:48.782693 5036 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.786756 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:48.789562 4876 leader_election.cc:341] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.788020 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.792164 4814 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:48.793892 5097 tablet_bootstrap.cc:654] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.796734 4930 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:48.807202 4882 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:48.812866 5129 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.813367 5129 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.813866 5129 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.827658 5093 tablet_bootstrap.cc:654] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:48.827939 4768 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.830228 5105 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:48.830735 5105 raft_consensus.cc:686] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:48.831964 5105 consensus_queue.cc:237] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.833338 5097 tablet_bootstrap.cc:492] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:48.833806 5097 ts_tablet_manager.cc:1397] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.008s
I20260806 04:06:48.847664 5129 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.849427 5336 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.856019 5129 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.870842 5336 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.871625 5336 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.873857 5092 tablet_bootstrap.cc:492] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:48.874466 5092 ts_tablet_manager.cc:1397] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.202s user 0.021s sys 0.019s
I20260806 04:06:48.874993 5097 raft_consensus.cc:348] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.885409 5097 raft_consensus.cc:374] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:48.885878 5097 raft_consensus.cc:729] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:48.888219 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:48.888197 5234 consensus_queue.cc:1035] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.889381 4816 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.889498 5337 consensus_queue.cc:1035] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:48.893445 5297 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.894177 5297 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.897132 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:48.897913 5297 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:48.887223 5097 consensus_queue.cc:260] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.874564 5336 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.918098 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.928714 4931 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
W20260806 04:06:48.931159 4876 leader_election.cc:341] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.933779 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:48.936120 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:48.937736 5129 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.938541 5129 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:48.941052 4992 leader_election.cc:341] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.946007 5160 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:48.947028 5160 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:48.937752 5336 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:48.948668 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:48.949512 4813 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:48.950109 5160 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.949190 5336 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:48.951304 5336 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.944798 5129 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:48.951514 4672 catalog_manager.cc:5495] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:48.957263 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.957702 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:48.960448 4882 leader_election.cc:341] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.961336 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:48.962379 4996 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:48.973690 5097 ts_tablet_manager.cc:1428] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.140s user 0.006s sys 0.009s
I20260806 04:06:48.974797 5097 tablet_bootstrap.cc:492] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
W20260806 04:06:48.973963 4876 leader_election.cc:341] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:48.978195 5263 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:49.004619 5263 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:49.005183 5263 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:48.999828 4816 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:48.987675 5036 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:49.013449 4876 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:49.024509 5297 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.037369 5389 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:49.025285 5297 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.046608 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:49.048938 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:49.130086 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 166386us
I20260806 04:06:49.052706 5297 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:49.077289 5336 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.134590 5092 raft_consensus.cc:348] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.135396 5092 raft_consensus.cc:374] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.135612 5092 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.136898 5092 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.138942 4882 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:49.139899 5092 ts_tablet_manager.cc:1428] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.255s user 0.009s sys 0.011s
I20260806 04:06:49.140095 5093 tablet_bootstrap.cc:492] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:49.140611 5093 ts_tablet_manager.cc:1397] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.441s user 0.019s sys 0.029s
I20260806 04:06:49.050935 5160 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.141835 5337 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 04:06:49.126561 4768 leader_election.cc:341] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.144599 5337 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:49.128091 5263 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.146215 5337 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.147909 5336 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:49.150419 5092 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:49.151989 5093 raft_consensus.cc:348] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.152662 5093 raft_consensus.cc:374] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.152920 5093 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.153554 5093 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.151002 5263 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:49.142628 5160 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.160859 5336 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:49.163079 5093 ts_tablet_manager.cc:1428] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.022s user 0.008s sys 0.001s
I20260806 04:06:49.165717 5097 tablet_bootstrap.cc:654] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.166193 5093 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:49.174283 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:49.175316 4817 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:49.175127 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:49.176663 4768 leader_election.cc:341] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.180383 5092 tablet_bootstrap.cc:654] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.181064 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:49.186335 5336 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:49.188169 5160 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:49.191486 4768 leader_election.cc:341] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.191270 5336 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:49.192138 4768 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:49.193475 5336 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:49.200145 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:49.201184 4813 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:49.203696 4992 leader_election.cc:341] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:49.204640 4992 leader_election.cc:341] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.205685 4996 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:49.210870 5297 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:49.211567 5297 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:49.212298 5297 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.233979 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:49.235031 4931 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.235803 5036 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.241770 4674 catalog_manager.cc:5495] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:49.255481 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:49.256425 4930 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:49.256865 5097 tablet_bootstrap.cc:492] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:49.257541 5097 ts_tablet_manager.cc:1397] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.283s user 0.024s sys 0.009s
I20260806 04:06:49.236248 5297 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.264947 4813 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:49.267102 5097 raft_consensus.cc:348] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.268057 5097 raft_consensus.cc:374] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.268481 5097 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.268361 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:49.275197 5097 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.276371 5297 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:49.289050 5093 tablet_bootstrap.cc:654] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.304891 5097 ts_tablet_manager.cc:1428] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.047s user 0.003s sys 0.006s
I20260806 04:06:49.306274 5097 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:49.317471 4996 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:49.310837 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:49.337216 5207 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:49.379101 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:49.379408 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:49.379958 4813 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.346491 5207 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:49.383637 5207 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.346405 5334 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:49.410221 4813 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:49.400007 4931 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:49.477932 5093 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:49.478374 5093 ts_tablet_manager.cc:1397] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.312s user 0.013s sys 0.014s
I20260806 04:06:49.487922 5105 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:06:49.490518 4768 leader_election.cc:341] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.495960 5105 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:49.475885 4996 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:49.499415 5213 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:49.500800 5213 raft_consensus.cc:686] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:49.504657 5092 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:49.505182 5092 ts_tablet_manager.cc:1397] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.355s user 0.007s sys 0.018s
I20260806 04:06:49.515678 5097 tablet_bootstrap.cc:654] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.503414 5213 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.521121 5093 raft_consensus.cc:348] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.520443 5092 raft_consensus.cc:348] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.521880 5093 raft_consensus.cc:374] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.522091 5092 raft_consensus.cc:374] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.522174 5093 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.522660 5092 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.523022 5093 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.523401 5092 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.525257 5093 ts_tablet_manager.cc:1428] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260806 04:06:49.526600 5092 ts_tablet_manager.cc:1428] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.021s user 0.007s sys 0.001s
I20260806 04:06:49.527840 5093 tablet_bootstrap.cc:492] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:49.527885 5092 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:49.523396 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:49.536619 5105 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:49.537317 5105 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:49.537901 5105 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.543332 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:49.544782 5160 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:49.545606 5160 raft_consensus.cc:686] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:49.547103 5160 consensus_queue.cc:237] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:49.563251 4992 leader_election.cc:341] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.603727 5234 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.608350 5234 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.612458 5234 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:49.618111 5093 tablet_bootstrap.cc:654] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.619778 4674 catalog_manager.cc:5495] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:49.637146 5092 tablet_bootstrap.cc:654] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.636657 5105 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.646054 4672 catalog_manager.cc:5495] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:49.647651 5392 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.648890 5392 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.652468 5392 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:49.646718 4768 leader_election.cc:341] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.656701 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:49.658739 5408 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.659969 5408 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.663345 5097 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:49.664211 5097 ts_tablet_manager.cc:1397] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.358s user 0.025s sys 0.030s
I20260806 04:06:49.660367 5105 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:49.671743 5408 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:49.662840 4672 catalog_manager.cc:5495] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:49.674352 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:49.780196 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:49.687629 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:49.786042 5097 raft_consensus.cc:348] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.787248 5097 raft_consensus.cc:374] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.787972 4931 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.792667 5336 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:49.787552 5097 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.794312 5097 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.708789 5337 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.807132 5337 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.816119 5097 ts_tablet_manager.cc:1428] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.152s user 0.013s sys 0.005s
I20260806 04:06:49.817492 5097 tablet_bootstrap.cc:492] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:49.822837 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:49.687448 4876 leader_election.cc:341] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:49.824473 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 137135us
W20260806 04:06:49.847640 4768 leader_election.cc:341] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.855538 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:49.856496 4814 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
W20260806 04:06:49.857903 4882 leader_election.cc:341] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.858469 4882 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:49.859011 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:49.859831 5234 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:49.851401 5337 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:49.870472 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:49.875154 5093 tablet_bootstrap.cc:492] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:49.875972 5093 ts_tablet_manager.cc:1397] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.348s user 0.045s sys 0.086s
I20260806 04:06:49.877403 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:49.881562 5092 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:49.882110 5092 ts_tablet_manager.cc:1397] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.354s user 0.015s sys 0.011s
I20260806 04:06:49.899470 5093 raft_consensus.cc:348] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.900683 5093 raft_consensus.cc:374] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.901294 5093 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.901489 5186 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:49.902372 5093 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.901539 5392 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:49.904842 5093 ts_tablet_manager.cc:1428] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20260806 04:06:49.908710 5234 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:49.909173 5093 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:49.909381 5234 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.912683 5234 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:49.887642 4931 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:49.919967 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:49.928231 4882 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:06:49.936915 4876 consensus_peers.cc:487] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:49.941402 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:49.944718 4814 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:49.942576 5186 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:49.946547 5186 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:49.947190 5186 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:49.954381 4992 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:06:49.956229 4876 leader_election.cc:341] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:49.964903 5263 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:49.971542 5092 raft_consensus.cc:348] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.976963 5092 raft_consensus.cc:374] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:49.978183 5092 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:49.974346 5093 tablet_bootstrap.cc:654] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:49.979614 5092 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:49.983978 5092 ts_tablet_manager.cc:1428] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.102s user 0.007s sys 0.002s
I20260806 04:06:49.985073 5092 tablet_bootstrap.cc:492] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:49.984995 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:50.004245 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:50.006812 5097 tablet_bootstrap.cc:654] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
W20260806 04:06:50.021747 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 123444us
W20260806 04:06:50.025666 4768 leader_election.cc:341] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.025983 4768 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:50.027896 5105 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:50.028100 4768 leader_election.cc:341] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.030862 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:50.021016 5408 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:06:50.050465 5160 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:50.131057 5093 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:50.131804 5093 ts_tablet_manager.cc:1397] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.223s user 0.023s sys 0.015s
I20260806 04:06:50.136186 4930 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.112066 5207 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.144534 5105 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.145489 5105 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.146198 5093 raft_consensus.cc:348] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.148274 5093 raft_consensus.cc:374] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.148528 5093 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.149019 5093 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.150460 5105 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.155469 5093 ts_tablet_manager.cc:1428] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s
I20260806 04:06:50.156719 5093 tablet_bootstrap.cc:492] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:50.167840 4876 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:50.150265 5207 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.180298 5034 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.146764 5160 raft_consensus.cc:686] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:06:50.182610 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 130042us
I20260806 04:06:50.192148 5186 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.215271 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:50.180279 5207 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:50.205989 5160 consensus_queue.cc:237] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.234004 5435 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:50.235023 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 150375us
I20260806 04:06:50.255580 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:50.255728 5297 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:50.247490 5186 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.271150 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:50.272243 4926 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
W20260806 04:06:50.279255 4768 leader_election.cc:341] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.280663 5092 tablet_bootstrap.cc:654] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
W20260806 04:06:50.281596 4768 consensus_peers.cc:487] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:50.286880 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:50.287802 4818 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.291595 5093 tablet_bootstrap.cc:654] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.293963 4768 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:50.300736 5392 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:50.301132 5392 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:50.301499 5392 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.301278 5334 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:50.327556 5297 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.328100 5297 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.328712 5097 tablet_bootstrap.cc:492] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:50.329263 5097 ts_tablet_manager.cc:1397] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.512s user 0.015s sys 0.010s
I20260806 04:06:50.330458 4672 catalog_manager.cc:5495] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:50.332547 5297 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:50.332765 4818 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:50.337075 5097 raft_consensus.cc:348] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.338382 5097 raft_consensus.cc:374] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.338835 5097 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.339869 5097 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.357694 5097 ts_tablet_manager.cc:1428] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20260806 04:06:50.359373 5097 tablet_bootstrap.cc:492] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
W20260806 04:06:50.356559 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 188867us
I20260806 04:06:50.355291 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:50.372280 4882 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:50.400133 5186 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:50.400934 5186 raft_consensus.cc:686] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:50.403914 5186 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.412034 5213 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:50.366273 5392 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.421352 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:50.426564 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:50.427589 4930 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.428675 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:50.430325 4768 leader_election.cc:341] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.431205 5392 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.433971 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:50.436239 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:50.440152 5093 tablet_bootstrap.cc:492] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:50.440606 5093 ts_tablet_manager.cc:1397] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.284s user 0.007s sys 0.014s
I20260806 04:06:50.441954 5032 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.449182 5093 raft_consensus.cc:348] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.450103 5093 raft_consensus.cc:374] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.450410 5093 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.451153 5093 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.456110 4930 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:50.460137 4768 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:50.462055 5093 ts_tablet_manager.cc:1428] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.021s user 0.003s sys 0.006s
I20260806 04:06:50.462909 5097 tablet_bootstrap.cc:654] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.463200 5392 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:50.463099 5093 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:50.466811 5392 raft_consensus.cc:686] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:50.469200 5392 consensus_queue.cc:237] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.490233 5092 tablet_bootstrap.cc:492] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:50.490881 5092 ts_tablet_manager.cc:1397] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.506s user 0.019s sys 0.011s
I20260806 04:06:50.495577 5092 raft_consensus.cc:348] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.496125 5092 raft_consensus.cc:374] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.496294 5092 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.496773 5092 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.502396 5092 ts_tablet_manager.cc:1428] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260806 04:06:50.503428 5092 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
W20260806 04:06:50.504771 4992 consensus_peers.cc:487] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 -> Peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Couldn't send request to peer 22fe557f852d4415a96fd683f970cc24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:50.505838 4992 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:06:50.508752 4876 leader_election.cc:341] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.518105 4674 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:50.545609 5297 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:50.549371 5093 tablet_bootstrap.cc:654] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.553344 4992 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:50.563433 5297 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:50.567816 5334 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.578037 5334 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.584713 5334 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.589591 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:50.592417 4926 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:50.595486 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:50.589654 5092 tablet_bootstrap.cc:654] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.604756 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:50.611094 4768 leader_election.cc:341] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.600502 5334 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:50.616031 5097 tablet_bootstrap.cc:492] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:50.616495 5097 ts_tablet_manager.cc:1397] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.257s user 0.029s sys 0.016s
I20260806 04:06:50.613351 5334 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:50.618781 5334 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.621369 5097 raft_consensus.cc:348] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.622080 5097 raft_consensus.cc:374] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.622337 5097 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.624788 5097 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.624801 5186 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.626909 5186 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.646107 5093 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:50.646776 5093 ts_tablet_manager.cc:1397] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.184s user 0.013s sys 0.015s
I20260806 04:06:50.657029 5097 ts_tablet_manager.cc:1428] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.040s user 0.001s sys 0.008s
I20260806 04:06:50.658098 5097 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:50.650907 5093 raft_consensus.cc:348] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.658975 5093 raft_consensus.cc:374] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.659263 5093 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.659927 5093 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.664465 5160 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.666980 5160 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.671424 5093 ts_tablet_manager.cc:1428] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.024s user 0.004s sys 0.003s
I20260806 04:06:50.672443 5093 tablet_bootstrap.cc:492] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:50.675201 5160 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.675843 5186 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.676486 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:50.677703 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:50.678113 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:50.678535 4882 leader_election.cc:341] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:50.681025 4768 leader_election.cc:341] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:50.685971 4876 leader_election.cc:341] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.688354 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:50.689025 4876 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:50.697997 5186 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:50.700944 4768 leader_election.cc:341] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.701452 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:50.725788 5392 consensus_queue.cc:1035] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:50.708491 5334 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.724079 5421 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:50.734989 5092 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:50.735570 5092 ts_tablet_manager.cc:1397] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.232s user 0.016s sys 0.033s
I20260806 04:06:50.736562 5034 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.738601 5092 raft_consensus.cc:348] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.739383 5092 raft_consensus.cc:374] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.739647 5092 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.740252 5092 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.744904 5097 tablet_bootstrap.cc:654] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.743618 5092 ts_tablet_manager.cc:1428] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260806 04:06:50.746359 5092 tablet_bootstrap.cc:492] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:50.749358 5093 tablet_bootstrap.cc:654] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.753747 5334 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.754558 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:50.759708 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:50.759624 4669 catalog_manager.cc:5495] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
W20260806 04:06:50.769200 4768 leader_election.cc:341] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.760378 4928 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.776326 5092 tablet_bootstrap.cc:654] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.783766 5439 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.784543 5439 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.789881 4928 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:50.791970 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:50.792258 5439 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.792899 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:50.810329 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:50.818564 5431 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:50.821123 4882 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:06:50.822153 4768 consensus_peers.cc:487] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:50.823220 5337 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:50.821170 5431 raft_consensus.cc:686] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:50.825448 5431 consensus_queue.cc:237] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.843202 5097 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:50.843879 5097 ts_tablet_manager.cc:1397] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.186s user 0.006s sys 0.018s
I20260806 04:06:50.848542 5097 raft_consensus.cc:348] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.849664 5097 raft_consensus.cc:374] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.849969 5097 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.850690 5097 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.862972 5093 tablet_bootstrap.cc:492] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:50.863512 5093 ts_tablet_manager.cc:1397] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.191s user 0.006s sys 0.018s
I20260806 04:06:50.869673 5093 raft_consensus.cc:348] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.874500 5097 ts_tablet_manager.cc:1428] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260806 04:06:50.875655 5097 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:50.876169 5093 raft_consensus.cc:374] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.876541 5093 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.877081 5093 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.879925 5092 tablet_bootstrap.cc:492] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:50.880398 5092 ts_tablet_manager.cc:1397] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.134s user 0.016s sys 0.014s
I20260806 04:06:50.890977 4674 catalog_manager.cc:5495] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:50.894874 5097 tablet_bootstrap.cc:654] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.903666 5093 ts_tablet_manager.cc:1428] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.040s user 0.004s sys 0.005s
I20260806 04:06:50.907892 5092 raft_consensus.cc:348] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.908684 5092 raft_consensus.cc:374] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.909034 5092 raft_consensus.cc:729] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.913419 5092 consensus_queue.cc:260] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.915385 5092 ts_tablet_manager.cc:1428] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.035s user 0.005s sys 0.004s
I20260806 04:06:50.915702 5097 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:50.916271 5097 ts_tablet_manager.cc:1397] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.013s
I20260806 04:06:50.922966 5337 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.938447 5337 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.940454 5337 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.932894 5092 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:50.942461 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:50.920521 5093 tablet_bootstrap.cc:492] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
W20260806 04:06:50.944077 4876 leader_election.cc:341] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.934281 5097 raft_consensus.cc:348] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.947165 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:50.947603 5097 raft_consensus.cc:374] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:06:50.948700 4882 leader_election.cc:341] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:50.949219 4882 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:50.950202 5093 tablet_bootstrap.cc:654] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.950466 5435 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:50.947991 5097 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.953361 5097 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:50.955345 4876 consensus_peers.cc:487] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:50.955564 5337 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:50.956488 5092 tablet_bootstrap.cc:654] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:50.956410 5337 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.960753 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:50.961134 5337 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:50.961750 4816 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:50.962604 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:50.963441 4882 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:50.964550 5034 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:50.965411 5337 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:50.965912 5337 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:50.966526 5337 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:50.972865 5097 ts_tablet_manager.cc:1428] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.056s user 0.003s sys 0.006s
I20260806 04:06:50.973874 5093 tablet_bootstrap.cc:492] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:50.974597 5093 ts_tablet_manager.cc:1397] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.003s
I20260806 04:06:50.973974 5097 tablet_bootstrap.cc:492] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:50.978626 5093 raft_consensus.cc:348] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.979581 5093 raft_consensus.cc:374] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:50.979831 5093 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:50.980609 5093 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:50.984747 5093 ts_tablet_manager.cc:1428] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.010s user 0.003s sys 0.005s
I20260806 04:06:50.986229 5093 tablet_bootstrap.cc:492] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:50.996125 5234 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:50.996774 5337 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.013162 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:51.011955 5337 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.013250 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:51.014226 5034 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.014698 4814 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.050724 4930 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.068060 5092 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:51.067842 4814 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:51.068673 5092 ts_tablet_manager.cc:1397] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.152s user 0.015s sys 0.025s
I20260806 04:06:51.068622 5234 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.074257 5234 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.075970 4882 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:51.077246 5435 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:51.078059 5435 raft_consensus.cc:686] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:51.079483 5435 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:51.069221 4768 consensus_peers.cc:487] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:51.084764 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:51.087415 4876 leader_election.cc:341] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.088303 5234 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.089895 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.099647 4816 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:51.102968 5097 tablet_bootstrap.cc:654] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.104689 5092 raft_consensus.cc:348] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.105501 5092 raft_consensus.cc:374] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.105906 5092 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.072176 5093 tablet_bootstrap.cc:654] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.106880 5092 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.106261 4882 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.109423 5234 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.109994 5234 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.110596 5234 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.117470 5160 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:51.121683 5097 tablet_bootstrap.cc:492] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:51.122224 5097 ts_tablet_manager.cc:1397] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.017s
I20260806 04:06:51.130492 5097 raft_consensus.cc:348] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.131309 5097 raft_consensus.cc:374] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.131614 5097 raft_consensus.cc:729] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.132341 5097 consensus_queue.cc:260] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.134250 5097 ts_tablet_manager.cc:1428] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20260806 04:06:51.130740 5092 ts_tablet_manager.cc:1428] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.057s user 0.010s sys 0.004s
I20260806 04:06:51.135290 5097 tablet_bootstrap.cc:492] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:51.138804 5092 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:51.140702 5186 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.134404 5034 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:51.160086 5186 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.164650 5263 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.178141 5263 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.182168 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:51.183410 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.185868 5263 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:51.187297 4996 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:51.187839 5186 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.190449 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:51.191054 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.195312 5263 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:51.196328 4882 leader_election.cc:341] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:51.197026 4876 leader_election.cc:341] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.197435 4876 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:51.198607 5186 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:51.206461 4669 catalog_manager.cc:5495] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:51.216271 5234 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.222374 5097 tablet_bootstrap.cc:654] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.222625 5092 tablet_bootstrap.cc:654] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.228453 5093 tablet_bootstrap.cc:492] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:51.229941 5093 ts_tablet_manager.cc:1397] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.244s user 0.013s sys 0.016s
I20260806 04:06:51.241990 5093 raft_consensus.cc:348] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.242950 5093 raft_consensus.cc:374] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.243228 5093 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.244608 5093 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.245726 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:51.246372 4814 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.246615 5093 ts_tablet_manager.cc:1428] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.016s user 0.003s sys 0.002s
I20260806 04:06:51.247514 5093 tablet_bootstrap.cc:492] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:51.255784 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:51.256124 5097 tablet_bootstrap.cc:492] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:51.256806 5097 ts_tablet_manager.cc:1397] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.023s
W20260806 04:06:51.256963 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:51.257294 4814 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
W20260806 04:06:51.257325 4876 leader_election.cc:341] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.257402 5234 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.258927 4882 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.260208 5234 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:51.260463 5092 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:51.260890 5234 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:51.261047 5092 ts_tablet_manager.cc:1397] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.122s user 0.010s sys 0.007s
I20260806 04:06:51.261426 5097 raft_consensus.cc:348] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.261788 5234 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.262531 5097 raft_consensus.cc:374] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.263012 5097 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.263682 5097 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.265988 5097 ts_tablet_manager.cc:1428] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260806 04:06:51.266598 5093 tablet_bootstrap.cc:654] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.269209 5097 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:51.270531 5092 raft_consensus.cc:348] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.271270 5092 raft_consensus.cc:374] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.271514 5092 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.272176 5092 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.275184 5097 tablet_bootstrap.cc:654] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.280296 5092 ts_tablet_manager.cc:1428] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260806 04:06:51.281452 5092 tablet_bootstrap.cc:492] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:51.285435 5034 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.287411 5092 tablet_bootstrap.cc:654] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.289229 5097 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:51.289727 5097 ts_tablet_manager.cc:1397] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.002s
I20260806 04:06:51.293805 5097 raft_consensus.cc:348] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.294879 5097 raft_consensus.cc:374] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.295179 5097 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.295948 5097 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.299296 5097 ts_tablet_manager.cc:1428] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260806 04:06:51.300266 5097 tablet_bootstrap.cc:492] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:51.304085 5092 tablet_bootstrap.cc:492] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:51.304612 5092 ts_tablet_manager.cc:1397] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20260806 04:06:51.309365 5092 raft_consensus.cc:348] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.310185 5092 raft_consensus.cc:374] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.310472 5092 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.311216 5092 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.313807 5092 ts_tablet_manager.cc:1428] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 04:06:51.316305 5092 tablet_bootstrap.cc:492] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:51.317473 5334 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.318241 5334 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.321631 5093 tablet_bootstrap.cc:492] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:51.322127 5093 ts_tablet_manager.cc:1397] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.007s
I20260806 04:06:51.322307 5097 tablet_bootstrap.cc:654] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.326470 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:51.331259 5093 raft_consensus.cc:348] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.332439 5093 raft_consensus.cc:374] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.332799 5093 raft_consensus.cc:729] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
W20260806 04:06:51.333215 4768 leader_election.cc:341] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.335374 5093 consensus_queue.cc:260] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.354698 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:51.358481 5431 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:51.360227 4669 catalog_manager.cc:5495] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
W20260806 04:06:51.364681 4768 leader_election.cc:341] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.365163 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.367625 5093 ts_tablet_manager.cc:1428] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.045s user 0.008s sys 0.009s
I20260806 04:06:51.369627 5093 tablet_bootstrap.cc:492] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:51.339493 5334 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.368018 5392 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:51.398538 5421 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.399207 5421 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.401602 5093 tablet_bootstrap.cc:654] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.401173 5421 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.405460 5097 tablet_bootstrap.cc:492] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:51.406045 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:51.407534 5092 tablet_bootstrap.cc:654] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.408964 5097 ts_tablet_manager.cc:1397] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.018s
I20260806 04:06:51.409277 4928 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:51.410617 4768 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:51.406399 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:51.412400 5392 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.416364 5392 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.417505 5392 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.415011 5036 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
W20260806 04:06:51.444731 4768 consensus_peers.cc:487] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:51.447645 5392 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.450372 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:51.451004 4928 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.454481 5097 raft_consensus.cc:348] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.461809 5097 raft_consensus.cc:374] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.462096 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:51.461196 5392 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.462818 5036 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.462312 5097 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.469139 4928 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:51.470528 5097 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.473449 4768 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:51.474822 5334 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:51.475329 5334 raft_consensus.cc:686] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:51.476142 5334 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.481091 5092 tablet_bootstrap.cc:492] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:51.481091 5093 tablet_bootstrap.cc:492] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:51.482272 5093 ts_tablet_manager.cc:1397] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.114s user 0.013s sys 0.013s
I20260806 04:06:51.481624 5092 ts_tablet_manager.cc:1397] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.165s user 0.021s sys 0.008s
I20260806 04:06:51.484530 5097 ts_tablet_manager.cc:1428] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.073s user 0.011s sys 0.003s
I20260806 04:06:51.486616 5092 raft_consensus.cc:348] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.487380 5092 raft_consensus.cc:374] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.487648 5092 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.488320 5092 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.483799 5036 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:51.491264 5093 raft_consensus.cc:348] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.491989 5093 raft_consensus.cc:374] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.492262 5093 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.492974 5093 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.495434 5234 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.496095 5097 tablet_bootstrap.cc:492] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:51.496335 5093 ts_tablet_manager.cc:1428] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260806 04:06:51.497241 5092 ts_tablet_manager.cc:1428] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20260806 04:06:51.498423 5092 tablet_bootstrap.cc:492] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:51.500269 4674 catalog_manager.cc:5495] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:51.495883 5234 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.503032 5093 tablet_bootstrap.cc:492] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:51.507135 5097 tablet_bootstrap.cc:654] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.502758 5036 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.513962 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.515156 4814 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:51.517104 4882 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:51.518388 5337 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.518988 5337 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.519573 5337 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.525843 5093 tablet_bootstrap.cc:654] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.533150 5092 tablet_bootstrap.cc:654] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.536010 5421 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.536937 5421 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.539949 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:51.540464 5421 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:51.542439 4876 leader_election.cc:341] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.542963 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:51.545698 4768 leader_election.cc:341] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.543510 5234 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.546075 5337 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.551419 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:51.553288 4768 leader_election.cc:341] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.553872 4768 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.553836 5234 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.555992 5421 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:51.557046 5234 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.561131 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:51.562284 4814 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.567553 5234 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.566347 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.573094 4816 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:51.573704 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:51.575862 4876 leader_election.cc:341] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.581137 4882 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.579383 5337 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.583424 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:51.590349 4876 leader_election.cc:341] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.583020 5337 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.596056 5337 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.598626 5337 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.607290 5234 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:51.614886 5092 tablet_bootstrap.cc:492] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:51.615406 5092 ts_tablet_manager.cc:1397] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.117s user 0.012s sys 0.018s
I20260806 04:06:51.616914 5435 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.617748 5435 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.620445 5092 raft_consensus.cc:348] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.621124 5092 raft_consensus.cc:374] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.621522 5092 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.623363 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:51.624111 5435 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:51.625352 4876 leader_election.cc:341] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.626142 5092 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.628247 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.628925 5092 ts_tablet_manager.cc:1428] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260806 04:06:51.629180 4816 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:51.630041 5092 tablet_bootstrap.cc:492] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:51.634624 5097 tablet_bootstrap.cc:492] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:51.635278 5097 ts_tablet_manager.cc:1397] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.148s user 0.022s sys 0.015s
I20260806 04:06:51.634827 4882 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.636302 5093 tablet_bootstrap.cc:492] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:51.636847 5093 ts_tablet_manager.cc:1397] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.134s user 0.009s sys 0.020s
I20260806 04:06:51.639954 5435 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.640049 5097 raft_consensus.cc:348] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.640743 5435 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.641255 5097 raft_consensus.cc:374] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.642158 5097 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.643362 5097 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.641477 5435 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.652120 5097 ts_tablet_manager.cc:1428] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260806 04:06:51.653681 5097 tablet_bootstrap.cc:492] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:51.655611 5093 raft_consensus.cc:348] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.656425 5093 raft_consensus.cc:374] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.648638 4814 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:51.656688 5093 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.652961 5092 tablet_bootstrap.cc:654] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.657560 5093 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.659711 5337 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.664940 5093 ts_tablet_manager.cc:1428] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.028s user 0.007s sys 0.003s
I20260806 04:06:51.668040 5334 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.668722 5337 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.669811 5097 tablet_bootstrap.cc:654] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.674980 5435 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.676321 5093 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:51.676819 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:51.682597 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:51.683635 4816 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.685395 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:51.686254 4818 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.674826 5234 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 04:06:51.689044 5213 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.689821 5213 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.693300 5093 tablet_bootstrap.cc:654] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.695731 5435 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:51.699425 4876 leader_election.cc:341] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.669159 5334 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.700170 4818 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:51.703092 5213 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:51.705699 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:51.706686 5032 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:51.706993 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.708722 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:51.708487 4818 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:51.709682 5392 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.710294 5392 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.710543 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:51.710938 5392 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.711665 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:51.712514 4930 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:51.713619 5213 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:51.713449 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:51.715036 4928 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:51.707295 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:51.702639 5334 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:51.725662 4876 leader_election.cc:341] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:51.714211 5213 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:51.726644 5213 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.731415 5263 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.732349 5263 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.749140 4882 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:51.790258 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:51.839694 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:51.867792 4930 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:51.867949 4814 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:51.827248 4816 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:51.808184 5392 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.871091 5093 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:51.871575 5093 ts_tablet_manager.cc:1397] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.201s user 0.017s sys 0.006s
I20260806 04:06:51.872129 5392 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:51.874629 5093 raft_consensus.cc:348] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.876055 5093 raft_consensus.cc:374] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.876900 5093 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.879940 5093 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.883893 5526 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:51.796248 5263 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:51.909162 5526 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.912258 5093 ts_tablet_manager.cc:1428] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.040s user 0.006s sys 0.002s
I20260806 04:06:51.912503 5526 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:51.913185 5093 tablet_bootstrap.cc:492] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:51.914621 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:51.915282 4930 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:51.940104 5160 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:51.943279 4930 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:51.945106 5097 tablet_bootstrap.cc:492] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:51.945613 5097 ts_tablet_manager.cc:1397] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.292s user 0.044s sys 0.050s
I20260806 04:06:51.948804 5213 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.949359 5097 raft_consensus.cc:348] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.950062 5097 raft_consensus.cc:374] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:51.950341 5097 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:51.950909 5213 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:51.950971 5097 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:51.953164 5093 tablet_bootstrap.cc:654] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:51.953331 5097 ts_tablet_manager.cc:1428] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260806 04:06:51.953537 5092 tablet_bootstrap.cc:492] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:51.953989 5092 ts_tablet_manager.cc:1397] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.324s user 0.010s sys 0.026s
I20260806 04:06:51.954483 5097 tablet_bootstrap.cc:492] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:51.956427 5092 raft_consensus.cc:348] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.001083 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:06:52.010303 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 266476us
I20260806 04:06:51.957182 5092 raft_consensus.cc:374] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:06:52.015373 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 112175us
I20260806 04:06:52.023264 5392 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.023633 5337 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.027575 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.033105 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:52.034948 5097 tablet_bootstrap.cc:654] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.036105 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:52.046897 4992 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:52.047797 5037 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.056954 5337 raft_consensus.cc:686] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:52.058279 4931 raft_consensus.cc:2379] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:52.058916 5337 consensus_queue.cc:237] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.055068 5092 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.056600 5392 raft_consensus.cc:686] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:52.068440 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.068035 5034 raft_consensus.cc:2379] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:52.067755 4882 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:52.069865 5392 consensus_queue.cc:237] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.073693 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:52.082109 5092 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.091454 5520 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.095446 5520 raft_consensus.cc:686] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:52.101682 5524 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:52.110639 5524 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:52.111207 5524 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.113755 5092 ts_tablet_manager.cc:1428] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.159s user 0.001s sys 0.006s
W20260806 04:06:52.115552 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 69015.1us
I20260806 04:06:52.118307 5093 tablet_bootstrap.cc:492] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:52.118813 5431 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.096784 5520 consensus_queue.cc:237] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.119560 5431 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.122503 5431 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.124811 5093 ts_tablet_manager.cc:1397] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.212s user 0.012s sys 0.001s
I20260806 04:06:52.128635 5092 tablet_bootstrap.cc:492] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:52.130123 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.133299 4925 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:52.147511 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:52.148566 5431 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:52.149138 5431 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:52.149623 5431 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.155660 4992 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.165016 5213 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:06:52.172648 5097 tablet_bootstrap.cc:492] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:52.173427 5097 ts_tablet_manager.cc:1397] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.219s user 0.020s sys 0.020s
I20260806 04:06:52.167037 4882 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:52.179268 5093 raft_consensus.cc:348] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.184868 5093 raft_consensus.cc:374] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.185755 5093 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.183104 5186 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.179291 4992 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.188171 5186 raft_consensus.cc:686] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:52.180932 5097 raft_consensus.cc:348] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.189318 5097 raft_consensus.cc:374] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.189850 5097 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.189543 5186 consensus_queue.cc:237] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.190871 5097 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.204090 5097 ts_tablet_manager.cc:1428] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.030s user 0.009s sys 0.001s
I20260806 04:06:52.197162 5524 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.202142 5092 tablet_bootstrap.cc:654] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.191129 5093 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.205675 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:52.214193 5093 ts_tablet_manager.cc:1428] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.089s user 0.009s sys 0.000s
I20260806 04:06:52.216094 5097 tablet_bootstrap.cc:492] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:52.217135 5431 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.223691 5337 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.237279 5234 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.238113 5234 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.242602 5524 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:52.243495 5234 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.244941 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:52.246901 4876 leader_election.cc:341] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.247630 5092 tablet_bootstrap.cc:492] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:52.248529 5092 ts_tablet_manager.cc:1397] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.120s user 0.012s sys 0.007s
I20260806 04:06:52.251103 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:52.251897 4817 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.253207 5093 tablet_bootstrap.cc:492] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:52.242533 5431 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.224419 5337 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.263899 5392 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:52.267560 5337 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.280305 5092 raft_consensus.cc:348] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.281122 5092 raft_consensus.cc:374] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.282735 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:52.283589 4925 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.281417 5092 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.286018 5092 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.288376 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:52.287113 4672 catalog_manager.cc:5495] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
W20260806 04:06:52.292440 4876 leader_election.cc:341] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.298466 5097 tablet_bootstrap.cc:654] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.305595 5213 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:52.308481 4768 leader_election.cc:341] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.281383 4817 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:52.310243 4996 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:52.311335 5528 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.311586 4925 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:52.311829 5528 raft_consensus.cc:686] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:52.312798 5528 consensus_queue.cc:237] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.331879 5037 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.334514 5092 ts_tablet_manager.cc:1428] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.086s user 0.006s sys 0.001s
I20260806 04:06:52.339566 5092 tablet_bootstrap.cc:492] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:52.352845 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:52.357828 4931 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.361977 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.367017 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.367319 5213 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.367945 5213 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.374964 5092 tablet_bootstrap.cc:654] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.356401 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:52.376500 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.376852 5234 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:52.365954 4814 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:52.374240 5213 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:52.382304 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.383312 4818 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:52.392001 4996 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:52.395074 5535 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:06:52.395649 5535 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:06:52.396656 4882 leader_election.cc:341] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.397367 4882 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.398885 5186 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:52.396701 5535 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:06:52.400030 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51752.6us
W20260806 04:06:52.413066 4768 leader_election.cc:341] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.402184 5213 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.386668 5337 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 04:06:52.436344 4931 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:52.437917 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:52.440701 5560 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.441018 5097 tablet_bootstrap.cc:492] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:52.441283 5560 raft_consensus.cc:686] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:52.441567 5097 ts_tablet_manager.cc:1397] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.226s user 0.025s sys 0.029s
I20260806 04:06:52.444496 5093 tablet_bootstrap.cc:654] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.429250 5213 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.445096 4882 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:52.461303 5337 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:52.442307 5560 consensus_queue.cc:237] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.463204 5337 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:52.463696 5337 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.464903 5213 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:52.467839 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.470063 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.470599 4817 raft_consensus.cc:2379] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:52.462082 5535 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.471760 5032 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260806 04:06:52.473258 4996 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.476231 5213 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.479007 5234 consensus_queue.cc:1046] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:52.480036 5234 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:06:52.479839 5097 raft_consensus.cc:348] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.480473 5234 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:52.481458 5234 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.484265 5234 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:06:52.484568 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:52.485006 5535 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:52.480602 5097 raft_consensus.cc:374] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.485584 5097 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.486333 5097 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.492115 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:52.504256 5092 tablet_bootstrap.cc:492] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:52.502854 4674 catalog_manager.cc:5495] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:52.506112 5097 ts_tablet_manager.cc:1428] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.064s user 0.008s sys 0.000s
I20260806 04:06:52.507269 4674 catalog_manager.cc:5495] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:52.504941 5092 ts_tablet_manager.cc:1397] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.166s user 0.017s sys 0.015s
I20260806 04:06:52.523716 5160 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:52.507565 5097 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:52.525262 5093 tablet_bootstrap.cc:492] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:52.525740 5093 ts_tablet_manager.cc:1397] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.293s user 0.013s sys 0.006s
I20260806 04:06:52.526880 5092 raft_consensus.cc:348] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.527904 5092 raft_consensus.cc:374] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.528224 5092 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
W20260806 04:06:52.519910 4876 consensus_peers.cc:487] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:52.541950 4931 raft_consensus.cc:2396] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 04:06:52.545653 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.547235 5524 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:06:52.531993 5097 tablet_bootstrap.cc:654] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.537438 5213 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:52.535182 5092 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.538594 5337 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.520016 4669 catalog_manager.cc:5495] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:52.563210 5092 ts_tablet_manager.cc:1428] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.039s user 0.007s sys 0.001s
I20260806 04:06:52.562129 5536 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:52.567402 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:52.568078 4817 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.528311 5093 raft_consensus.cc:348] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.568629 5093 raft_consensus.cc:374] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.568969 5093 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.569696 5093 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.571415 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:52.573800 4876 leader_election.cc:341] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.578552 5092 tablet_bootstrap.cc:492] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:52.590900 5093 ts_tablet_manager.cc:1428] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.065s user 0.009s sys 0.000s
W20260806 04:06:52.593187 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:52.602701 5093 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
W20260806 04:06:52.602959 4876 consensus_peers.cc:487] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:52.566872 5337 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.621299 5093 tablet_bootstrap.cc:654] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.622582 5092 tablet_bootstrap.cc:654] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.614887 4669 catalog_manager.cc:5495] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:52.625455 5097 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:52.625927 5097 ts_tablet_manager.cc:1397] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.119s user 0.029s sys 0.021s
I20260806 04:06:52.628785 4817 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:52.644637 5097 raft_consensus.cc:348] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.645344 4882 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:52.647022 5337 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:52.647894 5337 raft_consensus.cc:686] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:52.649765 5097 raft_consensus.cc:374] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.650054 5097 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.651086 5097 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.653398 5097 ts_tablet_manager.cc:1428] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20260806 04:06:52.654343 5092 tablet_bootstrap.cc:492] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:52.654861 5092 ts_tablet_manager.cc:1397] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.077s user 0.019s sys 0.013s
I20260806 04:06:52.655186 5097 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:52.657483 5092 raft_consensus.cc:348] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.658324 5092 raft_consensus.cc:374] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.658574 5092 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.658062 4672 catalog_manager.cc:5495] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:52.659328 5092 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.661636 5092 ts_tablet_manager.cc:1428] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:06:52.662031 5097 tablet_bootstrap.cc:654] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.663172 5092 tablet_bootstrap.cc:492] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:52.670017 5092 tablet_bootstrap.cc:654] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.669827 5234 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:52.676401 5097 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:52.677495 5097 ts_tablet_manager.cc:1397] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20260806 04:06:52.682889 5097 raft_consensus.cc:348] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.683629 5097 raft_consensus.cc:374] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.683887 5097 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.684890 5097 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.687463 5097 ts_tablet_manager.cc:1428] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260806 04:06:52.688544 5097 tablet_bootstrap.cc:492] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:52.691402 5186 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:52.693921 5520 consensus_queue.cc:1035] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:52.698370 5092 tablet_bootstrap.cc:492] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:52.649036 5337 consensus_queue.cc:237] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.698877 5092 ts_tablet_manager.cc:1397] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.003s
I20260806 04:06:52.699350 5097 tablet_bootstrap.cc:654] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.701447 5092 raft_consensus.cc:348] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.702198 5092 raft_consensus.cc:374] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.702463 5092 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.703083 5092 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.705335 5092 ts_tablet_manager.cc:1428] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:06:52.706243 5092 tablet_bootstrap.cc:492] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:52.688645 5093 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:52.707283 5093 ts_tablet_manager.cc:1397] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.105s user 0.015s sys 0.007s
I20260806 04:06:52.709821 5213 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.710402 5213 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.712348 5092 tablet_bootstrap.cc:654] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.715258 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.717638 5535 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:52.715368 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.720696 4931 raft_consensus.cc:2379] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:52.724409 5092 tablet_bootstrap.cc:492] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:52.724962 5092 ts_tablet_manager.cc:1397] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260806 04:06:52.726480 4992 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.727619 5092 raft_consensus.cc:348] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.728358 5092 raft_consensus.cc:374] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.728621 5092 raft_consensus.cc:729] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.729242 5092 consensus_queue.cc:260] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.730319 5528 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:52.754617 5528 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:52.755106 5524 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.755596 5524 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.757819 5092 ts_tablet_manager.cc:1428] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.032s user 0.008s sys 0.001s
I20260806 04:06:52.758109 5524 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:52.750640 5213 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:52.759053 5092 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:52.764760 5234 consensus_queue.cc:1035] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:52.762144 5093 raft_consensus.cc:348] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.768239 5093 raft_consensus.cc:374] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:52.769160 5093 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:52.770597 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.790503 5524 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:06:52.795197 4992 leader_election.cc:341] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.799908 5092 tablet_bootstrap.cc:654] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:52.799695 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:52.805307 5093 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:52.810933 4996 leader_election.cc:341] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:52.811575 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:52.814082 5528 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:52.858361 5435 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.881918 5093 ts_tablet_manager.cc:1428] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.173s user 0.007s sys 0.005s
I20260806 04:06:52.890539 5421 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.891073 5421 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.893177 5421 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.863068 5520 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.894387 5520 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.863039 5435 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.959218 5093 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:52.957741 5528 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:52.968909 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:52.965319 5435 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.969687 5528 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:52.972419 5520 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:52.973482 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.024784 4669 catalog_manager.cc:5495] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:53.028528 5421 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.044605 5421 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.090984 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.092638 5524 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.093362 5097 tablet_bootstrap.cc:492] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:53.093176 5524 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.093966 5097 ts_tablet_manager.cc:1397] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.406s user 0.054s sys 0.079s
I20260806 04:06:53.097929 5421 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.100008 5528 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:53.101248 5092 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:53.103106 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.102536 5092 ts_tablet_manager.cc:1397] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.344s user 0.022s sys 0.025s
W20260806 04:06:53.105641 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 76884.4us
W20260806 04:06:53.106269 4768 leader_election.cc:341] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.108906 5524 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:53.110333 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.113669 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.114425 5093 tablet_bootstrap.cc:654] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.114332 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:53.117147 4876 leader_election.cc:341] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.118815 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.120160 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.120784 4813 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:53.125324 5092 raft_consensus.cc:348] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.126741 5092 raft_consensus.cc:374] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.127022 5092 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.127666 5092 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:53.129092 4876 leader_election.cc:341] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.131851 5035 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:53.133620 4876 consensus_peers.cc:487] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:53.156369 5092 ts_tablet_manager.cc:1428] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.051s user 0.003s sys 0.003s
I20260806 04:06:53.142428 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.164863 4882 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.167725 5032 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.169024 5097 raft_consensus.cc:348] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.169763 5097 raft_consensus.cc:374] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.170030 5097 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.170616 5097 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.172991 5097 ts_tablet_manager.cc:1428] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.079s user 0.007s sys 0.007s
I20260806 04:06:53.173906 5097 tablet_bootstrap.cc:492] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:53.174345 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.186940 4996 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:53.159161 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:06:53.189500 4768 leader_election.cc:341] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.189644 4996 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:53.190003 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.195456 5092 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:53.199625 5234 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:53.200098 5234 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:53.200467 5234 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:53.209846 4768 leader_election.cc:341] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.238830 5536 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:06:53.270759 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 82653.3us
W20260806 04:06:53.283004 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 43976.1us
W20260806 04:06:53.285074 4882 leader_election.cc:341] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.285907 4882 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:06:53.296165 4768 leader_election.cc:341] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.296896 5093 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:53.296824 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.297432 5093 ts_tablet_manager.cc:1397] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.339s user 0.028s sys 0.008s
I20260806 04:06:53.285713 5524 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.312846 5234 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.304767 5337 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.322621 5234 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.317726 5560 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260806 04:06:53.313819 5612 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.369999 5097 tablet_bootstrap.cc:654] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.380673 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:53.384169 5092 tablet_bootstrap.cc:654] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
W20260806 04:06:53.393822 4876 leader_election.cc:341] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.348023 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:53.395099 5093 raft_consensus.cc:348] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.396960 5093 raft_consensus.cc:374] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.397215 5093 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.397750 5093 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.401877 4818 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:53.417517 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 100387us
I20260806 04:06:53.419505 5093 ts_tablet_manager.cc:1428] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.122s user 0.006s sys 0.006s
I20260806 04:06:53.434473 5524 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.421517 4818 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:53.435041 5524 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.435885 4882 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.437024 5524 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:53.440797 5605 consensus_queue.cc:1035] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260806 04:06:53.445302 5093 tablet_bootstrap.cc:492] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:53.446929 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.452638 5602 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.453152 5602 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.444765 5520 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:53.462438 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.463205 4817 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:53.465368 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:53.481865 5234 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:53.482503 5234 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:53.482978 5234 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.458818 5602 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.491933 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:53.494814 4876 leader_election.cc:341] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.492239 5234 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.473165 5520 raft_consensus.cc:686] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:53.505856 5535 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.505993 5520 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.510319 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:53.510643 5234 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:53.511596 4876 leader_election.cc:341] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.511615 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:53.513108 4818 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.532238 5035 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.539862 5560 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:53.546167 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.554370 4992 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:53.564227 5092 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:53.568351 5092 ts_tablet_manager.cc:1397] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.389s user 0.012s sys 0.018s
I20260806 04:06:53.565230 4818 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:53.566622 5097 tablet_bootstrap.cc:492] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:53.575516 5097 ts_tablet_manager.cc:1397] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.402s user 0.016s sys 0.025s
I20260806 04:06:53.579988 5097 raft_consensus.cc:348] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.587052 5097 raft_consensus.cc:374] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.587503 5097 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.595288 5097 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.598064 5620 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.598559 5620 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.602943 5620 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.581579 5528 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.603327 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.605214 5435 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:53.605687 5435 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:53.571717 5092 raft_consensus.cc:348] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.603996 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.603996 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:53.607407 4768 leader_election.cc:341] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:53.608196 4768 leader_election.cc:341] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.608584 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.610013 5160 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.611923 4669 catalog_manager.cc:5495] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:53.614300 5092 raft_consensus.cc:374] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.607049 5435 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.618554 5092 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.619457 5092 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.573577 5093 tablet_bootstrap.cc:654] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.621908 5097 ts_tablet_manager.cc:1428] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.046s user 0.005s sys 0.003s
I20260806 04:06:53.639077 5097 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:53.642704 5092 ts_tablet_manager.cc:1428] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.074s user 0.007s sys 0.000s
I20260806 04:06:53.656234 4674 catalog_manager.cc:5495] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:53.666195 5092 tablet_bootstrap.cc:492] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:53.673974 5093 tablet_bootstrap.cc:492] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:53.674535 5093 ts_tablet_manager.cc:1397] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.229s user 0.009s sys 0.004s
W20260806 04:06:53.682979 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:06:53.686090 5093 raft_consensus.cc:348] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.688297 5093 raft_consensus.cc:374] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.688570 5093 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.686753 5560 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.689237 5560 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.692495 5097 tablet_bootstrap.cc:654] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.693264 5560 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.703091 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.703996 4925 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:53.705745 5092 tablet_bootstrap.cc:654] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.705911 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:53.710486 4768 leader_election.cc:341] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.712778 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.718237 5097 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:53.718782 5097 ts_tablet_manager.cc:1397] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.005s
I20260806 04:06:53.720772 5334 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:53.721182 5334 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:53.722141 5334 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.724596 5092 tablet_bootstrap.cc:492] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:53.725111 5092 ts_tablet_manager.cc:1397] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.011s
I20260806 04:06:53.728736 5097 raft_consensus.cc:348] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.729022 5092 raft_consensus.cc:348] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.729514 5097 raft_consensus.cc:374] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.729806 5092 raft_consensus.cc:374] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.729861 5097 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.730244 5092 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.731082 5092 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.732343 5097 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.733484 5092 ts_tablet_manager.cc:1428] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 04:06:53.734615 5097 ts_tablet_manager.cc:1428] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20260806 04:06:53.735705 5097 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:53.736881 5334 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.738803 5092 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:53.741466 5097 tablet_bootstrap.cc:654] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.689189 5093 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.750533 5093 ts_tablet_manager.cc:1428] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.076s user 0.004s sys 0.002s
I20260806 04:06:53.758205 5092 tablet_bootstrap.cc:654] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.769712 5334 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.771471 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:53.772166 4925 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.775339 5093 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:53.781949 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:53.794217 4768 leader_election.cc:341] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.804788 5213 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.805366 5213 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.807365 5213 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:53.807907 5093 tablet_bootstrap.cc:654] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.810850 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.812753 5092 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:53.813419 5092 ts_tablet_manager.cc:1397] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.007s
W20260806 04:06:53.814576 4992 leader_election.cc:341] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.816148 5092 raft_consensus.cc:348] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.816860 5092 raft_consensus.cc:374] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.817162 5092 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.817495 4925 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:53.817830 5092 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.820046 5092 ts_tablet_manager.cc:1428] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:06:53.820931 5092 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:53.830107 5435 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.830690 5435 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.831171 5097 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:53.831645 5097 ts_tablet_manager.cc:1397] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.096s user 0.020s sys 0.008s
I20260806 04:06:53.832691 5435 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.833351 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.834012 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.834714 5035 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:06:53.835783 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:53.836760 5435 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:06:53.837108 5435 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:53.837450 5435 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:53.839576 5092 tablet_bootstrap.cc:654] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.844643 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:53.845950 5421 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:53.846441 5421 raft_consensus.cc:686] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:53.847368 5421 consensus_queue.cc:237] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.834009 5097 raft_consensus.cc:348] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.855010 5097 raft_consensus.cc:374] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.855383 5097 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.858035 5528 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.857918 5097 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.858615 5528 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.859711 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.860453 4813 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:53.861672 5528 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:53.862078 4996 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:53.863677 5213 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:53.856729 5435 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.864046 5213 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:53.864410 5213 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.868541 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.871080 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.871917 4813 raft_consensus.cc:2379] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:53.872874 4996 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:53.873857 5528 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.880774 5097 ts_tablet_manager.cc:1428] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.049s user 0.005s sys 0.002s
I20260806 04:06:53.885411 5097 tablet_bootstrap.cc:492] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:53.886449 5435 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.896211 5093 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:53.896670 5093 ts_tablet_manager.cc:1397] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.122s user 0.006s sys 0.014s
I20260806 04:06:53.899372 5093 raft_consensus.cc:348] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.901577 5337 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:53.902069 5337 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.903740 5337 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.904513 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:53.905344 4813 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:53.891615 4672 catalog_manager.cc:5495] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:53.907128 5093 raft_consensus.cc:374] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.907492 5093 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.886986 5528 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:53.886832 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:53.909436 5093 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.877355 5213 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.888669 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:53.911170 5032 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:06:53.912407 5213 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:53.913780 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.914350 4882 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:53.914433 5036 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:53.915329 5337 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:53.915769 5337 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:53.916148 5337 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.918856 5032 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:06:53.920966 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:53.924978 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:53.925660 4813 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.921011 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:53.925652 5602 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:06:53.930151 5602 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:06:53.930325 4992 leader_election.cc:341] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:53.931013 5602 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.937853 5093 ts_tablet_manager.cc:1428] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.041s user 0.002s sys 0.004s
I20260806 04:06:53.938913 5093 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:53.944659 5093 tablet_bootstrap.cc:654] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.945928 4813 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:53.947168 4996 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:53.948371 5528 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:53.951989 5528 raft_consensus.cc:686] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:53.953125 5528 consensus_queue.cc:237] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.949883 5092 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:53.957156 5092 ts_tablet_manager.cc:1397] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.136s user 0.009s sys 0.003s
I20260806 04:06:53.958935 5097 tablet_bootstrap.cc:654] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:53.960733 5092 raft_consensus.cc:348] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.961726 5092 raft_consensus.cc:374] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.962040 5092 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.928726 5337 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.963516 5092 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.969105 5334 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:06:53.969456 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:53.969594 5334 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.971642 5334 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.968703 5337 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:53.973405 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:53.974735 5032 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:53.975929 5097 tablet_bootstrap.cc:492] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:53.975772 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:53.976373 5097 ts_tablet_manager.cc:1397] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.004s
I20260806 04:06:53.976547 5036 raft_consensus.cc:2354] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3.
I20260806 04:06:53.977836 4768 leader_election.cc:398] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3.
I20260806 04:06:53.978232 4768 leader_election.cc:401] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:06:53.978853 5560 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:06:53.979003 5097 raft_consensus.cc:348] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.979710 5097 raft_consensus.cc:374] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.979961 5097 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.967720 5092 ts_tablet_manager.cc:1428] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260806 04:06:53.984833 5032 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:53.985376 5097 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.986052 4876 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:53.986835 5594 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:53.987105 5093 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:53.987280 5594 raft_consensus.cc:686] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:53.987533 5093 ts_tablet_manager.cc:1397] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.023s
I20260806 04:06:53.987715 5097 ts_tablet_manager.cc:1428] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20260806 04:06:53.988680 5097 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:53.988176 5594 consensus_queue.cc:237] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.990576 5093 raft_consensus.cc:348] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.991302 5093 raft_consensus.cc:374] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:53.991568 5093 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:53.992224 5093 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:53.994257 5093 ts_tablet_manager.cc:1428] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:06:53.996117 5093 tablet_bootstrap.cc:492] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:53.996805 5092 tablet_bootstrap.cc:492] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:54.001864 5560 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3.
I20260806 04:06:54.003129 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:54.003742 4813 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.010684 5093 tablet_bootstrap.cc:654] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.010040 4669 catalog_manager.cc:5495] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:54.015443 4674 catalog_manager.cc:5495] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:54.021041 5097 tablet_bootstrap.cc:654] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.023703 5092 tablet_bootstrap.cc:654] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.027302 4813 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:54.036760 5594 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.039786 5594 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.041826 5594 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.043318 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.045644 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:54.043787 4674 catalog_manager.cc:5495] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:54.046262 4817 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
W20260806 04:06:54.048271 4876 leader_election.cc:341] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.052299 5093 tablet_bootstrap.cc:492] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:54.052815 5093 ts_tablet_manager.cc:1397] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.057s user 0.014s sys 0.013s
I20260806 04:06:54.051761 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.054550 5594 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:54.054939 5594 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:54.055311 5594 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.055616 5097 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:54.056545 5097 ts_tablet_manager.cc:1397] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.002s
I20260806 04:06:54.056806 5560 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.057358 5560 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.059095 5097 raft_consensus.cc:348] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.061093 5097 raft_consensus.cc:374] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.061429 5097 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.061749 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:54.062456 4926 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:54.063902 5093 raft_consensus.cc:348] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.064554 5093 raft_consensus.cc:374] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.064790 5093 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.062114 5097 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.065352 5093 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.066632 5097 ts_tablet_manager.cc:1428] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 04:06:54.067451 5093 ts_tablet_manager.cc:1428] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260806 04:06:54.067588 5097 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:54.068740 5594 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.070689 5594 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.071821 5093 tablet_bootstrap.cc:492] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:54.071780 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:54.073011 5097 tablet_bootstrap.cc:654] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.073268 4768 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:06:54.073557 4876 leader_election.cc:341] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.074488 5334 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:54.074903 5334 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:54.074996 5528 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.075325 5334 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.075489 5528 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.077526 5528 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:54.077653 5093 tablet_bootstrap.cc:654] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.083635 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:54.084335 4926 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:54.087845 4992 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:54.086483 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:54.059648 5092 tablet_bootstrap.cc:492] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:54.091737 5092 ts_tablet_manager.cc:1397] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.101s user 0.014s sys 0.002s
I20260806 04:06:54.094352 5337 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.099035 5337 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.100618 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:54.102092 4813 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.107100 5560 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.108645 5337 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.110421 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.099758 5528 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:54.112159 5528 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:54.112524 5528 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.105027 5594 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260806 04:06:54.117043 5097 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:54.118306 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:54.118381 5435 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:54.122882 5097 ts_tablet_manager.cc:1397] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.010s
I20260806 04:06:54.115451 5092 raft_consensus.cc:348] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.125267 5092 raft_consensus.cc:374] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.125520 5092 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.126240 5092 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.128630 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.129633 5035 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.128630 4809 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:54.131055 4882 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:54.134451 5594 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 04:06:54.135316 4876 leader_election.cc:341] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.129076 4813 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
W20260806 04:06:54.140151 4876 consensus_peers.cc:487] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:54.138808 5594 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:54.144084 5594 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.149708 5092 ts_tablet_manager.cc:1428] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260806 04:06:54.150199 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.150971 5093 tablet_bootstrap.cc:492] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:54.151245 5092 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:54.151434 5093 ts_tablet_manager.cc:1397] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.013s
I20260806 04:06:54.151302 5435 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:54.152083 5435 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:54.156620 5435 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.157459 5093 raft_consensus.cc:348] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.158406 5093 raft_consensus.cc:374] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.158768 5093 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.153753 5334 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.161693 4817 raft_consensus.cc:2379] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:54.163403 5097 raft_consensus.cc:348] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.164109 5097 raft_consensus.cc:374] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.159499 5093 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.165406 5097 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.166114 5097 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.168970 5097 ts_tablet_manager.cc:1428] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20260806 04:06:54.169965 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:54.170725 4925 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.173238 5528 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.174108 5097 tablet_bootstrap.cc:492] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:54.177772 5594 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.180680 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:54.183374 4876 leader_election.cc:341] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.184435 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:54.186548 5032 raft_consensus.cc:2379] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:54.186671 5093 ts_tablet_manager.cc:1428] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260806 04:06:54.187014 5097 tablet_bootstrap.cc:654] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.187686 5093 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:54.188390 5334 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.181805 4925 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:54.189845 4768 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.190740 5528 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:54.191632 5421 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:54.192292 5421 raft_consensus.cc:686] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:54.194545 5602 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.195468 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:54.196667 4926 raft_consensus.cc:2379] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:06:54.195578 5602 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.193393 5421 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.199331 5097 tablet_bootstrap.cc:492] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:54.201030 5097 ts_tablet_manager.cc:1397] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
I20260806 04:06:54.192339 5594 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.209084 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:54.209748 4818 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.211020 5334 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.215214 5092 tablet_bootstrap.cc:654] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.211490 5334 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.198515 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:54.236876 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:54.240851 4816 raft_consensus.cc:2379] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 1: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:06:54.251297 4818 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:54.253743 5160 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.256016 5097 raft_consensus.cc:348] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.256933 5097 raft_consensus.cc:374] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.257252 5097 raft_consensus.cc:729] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.242439 5602 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.259806 4882 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.261374 5097 consensus_queue.cc:260] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.254348 5160 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.275429 5337 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:54.297868 5334 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.266052 5602 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:54.299921 5602 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:54.306769 5160 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.306212 5602 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.307149 4674 catalog_manager.cc:5495] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:54.340071 5093 tablet_bootstrap.cc:654] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.322206 5520 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.341127 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:54.341544 5520 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.342222 5535 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:06:54.343643 5520 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.380489 5097 ts_tablet_manager.cc:1428] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.179s user 0.024s sys 0.029s
I20260806 04:06:54.430581 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:54.445547 4818 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:54.452735 4882 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:54.454617 5234 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:54.455016 5234 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:54.455317 5234 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.385222 5030 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:06:54.391942 5683 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.379396 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.380874 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.434381 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:54.467075 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:54.467753 4669 catalog_manager.cc:5495] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:54.482524 5097 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:54.481022 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.486414 5234 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.488054 4931 raft_consensus.cc:2379] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:54.491703 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:54.492372 4813 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.470311 5683 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.522164 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:54.545746 5234 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.486034 5034 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:54.552066 5092 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
W20260806 04:06:54.554003 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 59886.5us
I20260806 04:06:54.556011 5092 ts_tablet_manager.cc:1397] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.405s user 0.022s sys 0.025s
I20260806 04:06:54.565271 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.566148 5037 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:54.555821 5435 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.013s
I20260806 04:06:54.584955 5092 raft_consensus.cc:348] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.586450 5092 raft_consensus.cc:374] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:06:54.586721 4768 leader_election.cc:341] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.586791 5092 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.588863 5092 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:54.589155 4768 consensus_peers.cc:487] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:54.590327 4768 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:54.595883 5092 ts_tablet_manager.cc:1428] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20260806 04:06:54.598585 4818 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:06:54.603778 5234 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:54.596881 4927 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:06:54.640419 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 79889.3us
I20260806 04:06:54.644469 5683 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.646608 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:06:54.649516 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 55544.9us
I20260806 04:06:54.655043 5093 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:54.655620 5093 ts_tablet_manager.cc:1397] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.468s user 0.019s sys 0.007s
I20260806 04:06:54.638612 4813 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:54.683346 4882 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.687381 5337 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:54.687841 5337 raft_consensus.cc:686] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:54.688549 5337 consensus_queue.cc:237] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.671167 5560 raft_consensus.cc:2741] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 04:06:54.657337 5528 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260806 04:06:54.706321 5093 raft_consensus.cc:348] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.707178 5093 raft_consensus.cc:374] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.709252 5093 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.711395 5093 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.691804 5520 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:54.679236 5594 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
W20260806 04:06:54.712746 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 114455us
I20260806 04:06:54.718325 5093 ts_tablet_manager.cc:1428] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.062s user 0.005s sys 0.001s
W20260806 04:06:54.716269 4768 leader_election.cc:341] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.719240 4768 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.725353 5421 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:54.727298 5690 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:54.729423 5097 tablet_bootstrap.cc:654] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.729094 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:54.736721 5704 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:54.744530 5093 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:54.754606 5093 tablet_bootstrap.cc:654] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.762310 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.763234 5026 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 0.
I20260806 04:06:54.752734 5715 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.764240 5715 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.766315 5715 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.770498 5435 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:54.788923 5092 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
W20260806 04:06:54.791965 4768 leader_election.cc:341] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.795624 5092 tablet_bootstrap.cc:654] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.799772 5602 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:06:54.802510 4876 consensus_peers.cc:487] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:54.858014 5337 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:54.861218 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:54.866498 4768 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:54.867875 5715 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:54.868383 5715 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:54.868769 5715 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:54.869640 5092 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:54.870160 5092 ts_tablet_manager.cc:1397] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.000s
W20260806 04:06:54.874764 4876 consensus_peers.cc:487] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 -> Peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Couldn't send request to peer 99690415a7b3441fbdb5e943667556a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:54.882573 5097 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:54.883100 5097 ts_tablet_manager.cc:1397] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.413s user 0.029s sys 0.021s
I20260806 04:06:54.886256 5092 raft_consensus.cc:348] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.894018 5097 raft_consensus.cc:348] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.897290 5097 raft_consensus.cc:374] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.897354 5092 raft_consensus.cc:374] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.897644 5097 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.897789 5092 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:54.898406 5097 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.898519 5092 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.901008 5097 ts_tablet_manager.cc:1428] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260806 04:06:54.901065 5092 ts_tablet_manager.cc:1428] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20260806 04:06:54.902024 5097 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:54.903134 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:54.905036 5093 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:54.905575 5093 ts_tablet_manager.cc:1397] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.166s user 0.014s sys 0.046s
I20260806 04:06:54.907841 5097 tablet_bootstrap.cc:654] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:54.902024 5092 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:54.910679 4669 catalog_manager.cc:5495] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:54.914294 4669 catalog_manager.cc:5495] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:54.922888 5093 raft_consensus.cc:348] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.923745 5093 raft_consensus.cc:374] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:54.924064 5093 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
W20260806 04:06:54.926060 4768 leader_election.cc:341] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.950193 5093 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.956835 5093 ts_tablet_manager.cc:1428] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.051s user 0.008s sys 0.000s
I20260806 04:06:54.960085 5560 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:54.960894 5560 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:54.960170 5093 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
W20260806 04:06:54.966734 4768 leader_election.cc:341] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:54.967216 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:54.971539 5421 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:54.972805 5560 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:54.973778 5560 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:54.983588 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:06:54.987078 4768 leader_election.cc:341] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.003645 5092 tablet_bootstrap.cc:654] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.007947 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:55.012934 4744 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260806 04:06:55.013871 4768 leader_election.cc:341] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.014676 4768 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:55.024514 5435 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.025161 5435 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.030818 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.028348 5435 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:06:55.033025 4882 leader_election.cc:341] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.038043 5560 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:55.051296 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:55.055763 4876 leader_election.cc:341] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.056206 4876 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:55.053069 5097 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:55.062011 5097 ts_tablet_manager.cc:1397] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.160s user 0.009s sys 0.008s
I20260806 04:06:55.062640 5093 tablet_bootstrap.cc:654] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.067648 5712 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:55.054437 5692 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20260806 04:06:55.045722 5715 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.077062 5715 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:55.101300 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:55.102481 5035 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.104923 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:55.105953 4927 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.116904 5035 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:55.137290 5092 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.151262 5092 ts_tablet_manager.cc:1397] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.249s user 0.012s sys 0.015s
I20260806 04:06:55.143435 4768 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.130564 5097 raft_consensus.cc:348] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.161260 5097 raft_consensus.cc:374] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.161696 5097 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.169004 5092 raft_consensus.cc:348] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.170032 5092 raft_consensus.cc:374] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.170431 5092 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.172122 5092 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.155500 5560 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:55.162572 5097 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.180562 5097 ts_tablet_manager.cc:1428] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.118s user 0.005s sys 0.006s
I20260806 04:06:55.183413 4927 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:06:55.183995 5097 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: Bootstrap starting.
I20260806 04:06:55.186631 5092 ts_tablet_manager.cc:1428] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.035s user 0.004s sys 0.004s
I20260806 04:06:55.187749 5092 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.178946 5560 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:06:55.191220 5560 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.197996 5535 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.202203 5535 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.204630 5535 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.205926 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:55.216037 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.216804 4818 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
W20260806 04:06:55.218808 4992 leader_election.cc:341] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.222020 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:55.224325 5535 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:55.224964 5535 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:55.225693 5535 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.232398 5034 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.247838 5092 tablet_bootstrap.cc:654] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.242754 5036 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.253464 5700 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:55.260100 5093 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:55.260644 5093 ts_tablet_manager.cc:1397] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.301s user 0.021s sys 0.013s
I20260806 04:06:55.266026 4669 catalog_manager.cc:5495] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:55.276818 5097 tablet_bootstrap.cc:654] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.281704 5092 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.284327 5092 ts_tablet_manager.cc:1397] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.097s user 0.015s sys 0.011s
I20260806 04:06:55.290998 5712 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:06:55.304749 5093 raft_consensus.cc:348] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.305599 5093 raft_consensus.cc:374] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.305895 5093 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.307022 5093 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.287026 5535 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.312369 5092 raft_consensus.cc:348] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.313148 5092 raft_consensus.cc:374] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.313436 5092 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.327566 5092 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.366863 5092 ts_tablet_manager.cc:1428] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.067s user 0.006s sys 0.003s
I20260806 04:06:55.368381 5093 ts_tablet_manager.cc:1428] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.107s user 0.003s sys 0.006s
I20260806 04:06:55.361649 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:55.381193 4817 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.381623 5706 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.369818 5097 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: No bootstrap required, opened a new log
I20260806 04:06:55.383523 5097 ts_tablet_manager.cc:1397] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent bootstrapping tablet: real 0.201s user 0.008s sys 0.016s
I20260806 04:06:55.382993 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:06:55.394987 4992 leader_election.cc:341] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.398218 5093 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:55.399843 5092 tablet_bootstrap.cc:492] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.382405 5706 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.360148 5535 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.410697 5706 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:55.414280 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:55.416057 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:55.416145 5097 raft_consensus.cc:348] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.416930 5097 raft_consensus.cc:374] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.417198 5097 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.417814 5097 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:55.418974 4768 leader_election.cc:341] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:06:55.421418 4768 leader_election.cc:341] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.421782 4768 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:55.422995 5560 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:55.427860 5097 ts_tablet_manager.cc:1428] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260806 04:06:55.428968 4817 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:55.430233 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:06:55.430968 5535 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:55.431869 5535 raft_consensus.cc:686] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:55.432631 5535 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.451546 5092 tablet_bootstrap.cc:654] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.457792 5700 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.458328 5700 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.460341 5700 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:55.462167 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.462873 4817 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:55.465931 4882 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.466704 5700 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:55.467084 5700 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:55.468895 5093 tablet_bootstrap.cc:654] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.472434 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:06:55.474167 4876 leader_election.cc:341] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.467406 5700 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.477757 4669 catalog_manager.cc:5495] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:55.481904 5093 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:55.482156 5092 tablet_bootstrap.cc:492] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.482371 5093 ts_tablet_manager.cc:1397] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.008s
I20260806 04:06:55.482697 5092 ts_tablet_manager.cc:1397] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.083s user 0.015s sys 0.000s
I20260806 04:06:55.483788 5700 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.484889 5093 raft_consensus.cc:348] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.485577 5093 raft_consensus.cc:374] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.485432 5092 raft_consensus.cc:348] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.486189 5092 raft_consensus.cc:374] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.486460 5092 raft_consensus.cc:729] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.486469 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:55.486945 5700 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:55.487200 4817 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.488209 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:06:55.489439 4876 leader_election.cc:341] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.487211 5092 consensus_queue.cc:260] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.492002 5092 ts_tablet_manager.cc:1428] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20260806 04:06:55.492930 5092 tablet_bootstrap.cc:492] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.493394 4817 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:55.485880 5093 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.494921 4882 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:06:55.494845 5093 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.496131 5520 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:55.496217 5535 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.496616 5520 raft_consensus.cc:686] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:55.496824 5535 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.497543 5093 ts_tablet_manager.cc:1428] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260806 04:06:55.497501 5520 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.498317 5092 tablet_bootstrap.cc:654] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.498739 5093 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:55.500965 5535 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.501981 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:55.502610 4924 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:55.503615 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:06:55.504426 5535 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:55.504909 5535 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:55.505435 5535 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.510366 5093 tablet_bootstrap.cc:654] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.515648 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.516642 4817 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:55.535470 5093 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:55.536022 5093 ts_tablet_manager.cc:1397] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.008s
I20260806 04:06:55.536273 5535 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.540213 5093 raft_consensus.cc:348] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.541254 5093 raft_consensus.cc:374] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.541568 5093 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.541567 5535 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.541507 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:55.542574 4817 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.542630 5093 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.542833 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:55.543952 4924 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.545051 5093 ts_tablet_manager.cc:1428] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 04:06:55.546169 5093 tablet_bootstrap.cc:492] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24: Bootstrap starting.
I20260806 04:06:55.552065 4817 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:55.554229 5092 tablet_bootstrap.cc:492] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.555380 5092 ts_tablet_manager.cc:1397] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.063s user 0.020s sys 0.016s
I20260806 04:06:55.556568 4996 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.556998 5093 tablet_bootstrap.cc:654] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.558148 5535 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:55.559576 5535 raft_consensus.cc:686] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:55.562124 5092 raft_consensus.cc:348] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.565084 5092 raft_consensus.cc:374] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.565742 5092 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.562455 5535 consensus_queue.cc:237] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.564253 4924 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:55.567046 5092 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.573731 5092 ts_tablet_manager.cc:1428] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260806 04:06:55.575635 5092 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.579142 5093 tablet_bootstrap.cc:492] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24: No bootstrap required, opened a new log
I20260806 04:06:55.579654 5093 ts_tablet_manager.cc:1397] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.017s
I20260806 04:06:55.582858 5093 raft_consensus.cc:348] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.583806 5093 raft_consensus.cc:374] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.584141 5093 raft_consensus.cc:729] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.590746 5092 tablet_bootstrap.cc:654] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.585197 5093 consensus_queue.cc:260] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.599448 4669 catalog_manager.cc:5495] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:55.607223 5092 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.607625 5093 ts_tablet_manager.cc:1428] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260806 04:06:55.607954 5092 ts_tablet_manager.cc:1397] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.006s
I20260806 04:06:55.611986 5092 raft_consensus.cc:348] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.612952 5092 raft_consensus.cc:374] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.613309 5092 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.614179 5092 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.616652 5092 ts_tablet_manager.cc:1428] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:06:55.621522 5092 tablet_bootstrap.cc:492] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.628023 4669 catalog_manager.cc:5495] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:55.628713 5692 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:55.630366 5092 tablet_bootstrap.cc:654] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.648758 5092 tablet_bootstrap.cc:492] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.649400 5092 ts_tablet_manager.cc:1397] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.013s
I20260806 04:06:55.649469 5602 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.650003 5602 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.652369 5602 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:55.653980 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.654729 4809 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 0.
I20260806 04:06:55.657235 5683 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:55.657410 4882 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.658308 5602 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:55.658700 5602 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:55.659027 5602 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.659508 5092 raft_consensus.cc:348] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.660319 5092 raft_consensus.cc:374] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.660610 5092 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.661317 5092 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.663789 5092 ts_tablet_manager.cc:1428] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 04:06:55.666926 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:55.667474 5092 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.667804 5535 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.668242 5535 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:55.668540 4876 leader_election.cc:341] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:06:55.670338 5535 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.670903 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.673221 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:55.674948 4992 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:55.675782 5713 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:55.681507 5092 tablet_bootstrap.cc:654] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.687640 5602 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.690302 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:55.690909 4809 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.727155 5092 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.727900 5092 ts_tablet_manager.cc:1397] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.036s
I20260806 04:06:55.730693 5092 raft_consensus.cc:348] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.731564 5092 raft_consensus.cc:374] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.731945 5092 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.732710 5092 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.735417 5092 ts_tablet_manager.cc:1428] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:06:55.737106 5092 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Bootstrap starting.
I20260806 04:06:55.738034 5713 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.738883 5713 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.744746 5092 tablet_bootstrap.cc:654] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Neither blocks nor log segments found. Creating new log.
I20260806 04:06:55.751494 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.752233 4818 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:55.773655 5092 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: No bootstrap required, opened a new log
I20260806 04:06:55.773747 4996 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.774418 5092 ts_tablet_manager.cc:1397] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.010s
I20260806 04:06:55.774612 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:55.775489 4928 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 0.
I20260806 04:06:55.779757 5030 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.773545 5713 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.777832 5535 raft_consensus.cc:2781] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:06:55.781881 5535 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:06:55.781631 5092 raft_consensus.cc:348] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.782492 5092 raft_consensus.cc:374] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:06:55.782450 5535 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.782969 5092 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Initialized, Role: FOLLOWER
I20260806 04:06:55.783833 5092 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.786686 5092 ts_tablet_manager.cc:1428] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260806 04:06:55.791471 5036 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.813287 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:06:55.814152 5034 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.801368 5602 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:55.820611 5535 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:55.822937 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 68475.6us
I20260806 04:06:55.823846 5702 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:55.825863 5702 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.831773 5702 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.815402 4809 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:55.848652 4882 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.831821 5034 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:06:55.851251 5770 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:55.859670 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.861299 4817 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 1: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:06:55.864390 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:06:55.865198 4817 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.863916 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:55.867635 4992 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:55.856324 5770 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:06:55.875229 4817 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:55.877354 4996 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:06:55.879329 5535 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 1 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:55.879926 5535 raft_consensus.cc:2781] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:06:55.871228 5702 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.885929 5770 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:55.863957 5713 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260806 04:06:55.880393 5535 raft_consensus.cc:686] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:06:55.909415 5535 consensus_queue.cc:237] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:06:55.948051 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 69215.2us
I20260806 04:06:55.949790 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:06:55.954388 4924 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:55.967051 4672 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:55.993541 5770 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:06:56.002738 4669 catalog_manager.cc:5495] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:06:56.003057 5770 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:56.010886 5770 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:56.013418 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:55.995931 5702 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:56.015597 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:06:56.023217 5773 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.023566 4924 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:06:56.035871 4882 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:56.036986 5773 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:06:56.059266 4924 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:56.062237 5700 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:56.069954 5702 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.109670 5702 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.121310 5702 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.179983 5034 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:06:56.187055 5713 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:06:56.187621 5713 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:06:56.194162 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:06:56.194766 5713 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:06:56.195824 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:06:56.198743 4992 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:06:56.199466 5713 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:06:56.334440 5700 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.345625 5700 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:06:56.399094 5713 consensus_queue.cc:1035] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.414008 5713 consensus_queue.cc:1035] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:06:56.475414 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:06:56.501856 5800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:06:56.502564 5804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:56.502704 3041 server_base.cc:1034] running on GCE node
W20260806 04:06:56.503254 5802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:06:56.506465 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:06:56.506846 3041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:06:56.507020 3041 hybrid_clock.cc:648] HybridClock initialized: now 1785989216507008 us; error 0 us; skew 500 ppm
I20260806 04:06:56.507709 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:06:56.516124 3041 webserver.cc:460] Webserver started at http://127.2.248.68:44515/ using document root <none> and password file <none>
I20260806 04:06:56.516701 3041 fs_manager.cc:359] Metadata directory not provided
I20260806 04:06:56.516963 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:06:56.517376 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:06:56.519157 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-3-root/instance:
uuid: "d7c76b6dcd8945cd818b8107caf4a174"
format_stamp: "Formatted at 2026-08-06 04:06:56 on dist-test-slave-3kk6"
I20260806 04:06:56.525388 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 04:06:56.529255 5811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:56.530133 3041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:06:56.530441 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-3-root
uuid: "d7c76b6dcd8945cd818b8107caf4a174"
format_stamp: "Formatted at 2026-08-06 04:06:56 on dist-test-slave-3kk6"
I20260806 04:06:56.530798 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:06:56.552022 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:06:56.554915 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:06:56.556716 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:06:56.566576 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:06:56.566927 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 04:06:56.567219 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:06:56.567400 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:06:56.666728 3041 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.68:38873
I20260806 04:06:56.666893 5901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.68:38873 every 8 connection(s)
I20260806 04:06:56.677979 3041 auto_leader_rebalancer-test.cc:413] add a tserver: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:06:56.692828 5902 heartbeater.cc:347] Connected to a master server at 127.2.248.126:44289
I20260806 04:06:56.693246 5902 heartbeater.cc:464] Registering TS with master...
I20260806 04:06:56.693910 5902 heartbeater.cc:511] Master 127.2.248.126:44289 requested a full tablet report, sending...
I20260806 04:06:56.695837 4672 ts_manager.cc:194] Registered new tserver with Master: d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:06:56.697757 4672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56998
I20260806 04:06:57.190402 4817 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.197266 5034 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.198838 5913 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:06:57.199846 4924 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.201383 5913 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:57.207562 5692 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.213577 5034 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:06:57.216024 4770 consensus_peers.cc:487] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4219680d51af4d0587436328294ba316. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:57.222553 4924 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.241570 4672 catalog_manager.cc:5495] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:57.265849 5032 consensus_queue.cc:237] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.272609 4928 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.273916 5702 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:57.275113 4817 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.276513 5702 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:57.287295 5713 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:06:57.291853 4995 consensus_peers.cc:487] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5666124a588f45948039fcd27abc9cf4. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:57.293859 4924 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.298151 4814 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.313792 4817 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.315907 4669 catalog_manager.cc:5495] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.345441 4928 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.347661 5913 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:06:57.348625 4770 consensus_peers.cc:487] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5088de56cbc4281b755364a8fddd084. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:57.351835 5032 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.353523 5913 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:06:57.359201 5692 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.362530 4924 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.379647 5034 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.387648 5032 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.390905 4672 catalog_manager.cc:5495] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:06:57.397336 4995 consensus_peers.cc:487] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98064c6d64f74659bfa102bd5a341635. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:06:57.399189 4924 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.400583 4817 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:06:57.408883 5713 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:57.410753 5713 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:06:57.426721 5702 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.430953 4924 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.433351 4817 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:06:57.453742 4672 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:06:57.645439 5946 ts_tablet_manager.cc:927] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:06:57.651763 5946 tablet_copy_client.cc:316] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:06:57.676918 4832 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 4219680d51af4d0587436328294ba316 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:06:57.678061 4832 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 4219680d51af4d0587436328294ba316 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-4219680d51af4d0587436328294ba316
I20260806 04:06:57.692122 4832 tablet_copy_source_session.cc:205] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:06:57.698845 5946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4219680d51af4d0587436328294ba316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:57.700762 5902 heartbeater.cc:503] Master 127.2.248.126:44289 was elected leader, sending a full tablet report...
I20260806 04:06:57.718626 5950 ts_tablet_manager.cc:927] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:57.725550 5950 tablet_copy_client.cc:316] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:06:57.745355 5946 tablet_copy_client.cc:791] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:06:57.746559 5946 tablet_copy_client.cc:655] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:06:57.757428 5946 tablet_copy_client.cc:531] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:06:57.760619 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 5666124a588f45948039fcd27abc9cf4 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:06:57.761482 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 5666124a588f45948039fcd27abc9cf4 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-5666124a588f45948039fcd27abc9cf4
I20260806 04:06:57.778086 5946 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:06:57.780341 5049 tablet_copy_source_session.cc:205] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:06:57.786049 5950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5666124a588f45948039fcd27abc9cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:57.811731 5950 tablet_copy_client.cc:791] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:06:57.812520 5950 tablet_copy_client.cc:655] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:06:57.819597 5950 tablet_copy_client.cc:531] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:06:57.824332 5952 ts_tablet_manager.cc:927] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:06:57.836966 5950 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:06:57.854887 5946 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:06:57.856210 5946 tablet_bootstrap.cc:492] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:06:57.856269 5952 tablet_copy_client.cc:316] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:06:57.868990 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 98064c6d64f74659bfa102bd5a341635 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:06:57.869953 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 98064c6d64f74659bfa102bd5a341635 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-98064c6d64f74659bfa102bd5a341635
I20260806 04:06:57.874642 5946 ts_tablet_manager.cc:1397] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.097s user 0.025s sys 0.034s
I20260806 04:06:57.891562 5049 tablet_copy_source_session.cc:205] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:06:57.899040 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98064c6d64f74659bfa102bd5a341635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:57.903120 5946 raft_consensus.cc:348] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.917119 5946 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:06:57.919948 5946 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.947583 5950 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:06:57.950688 5950 tablet_bootstrap.cc:492] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:06:57.951872 5950 ts_tablet_manager.cc:1397] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.115s user 0.033s sys 0.021s
I20260806 04:06:57.952795 5946 ts_tablet_manager.cc:1428] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.078s user 0.013s sys 0.005s
I20260806 04:06:57.953527 5952 tablet_copy_client.cc:791] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:06:57.962040 4832 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-4219680d51af4d0587436328294ba316 received from {username='slave'} at 127.0.0.1:53354
I20260806 04:06:57.962615 4832 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-4219680d51af4d0587436328294ba316 on tablet 4219680d51af4d0587436328294ba316 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:06:57.955336 5952 tablet_copy_client.cc:655] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:06:57.972078 5952 tablet_copy_client.cc:531] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:06:57.985275 5950 raft_consensus.cc:348] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.987638 5950 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:06:57.989013 5950 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:57.994109 5950 ts_tablet_manager.cc:1428] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.042s user 0.008s sys 0.002s
I20260806 04:06:57.995265 5965 ts_tablet_manager.cc:927] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:06:57.999162 5952 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:06:58.002722 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-5666124a588f45948039fcd27abc9cf4 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:06:58.003262 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-5666124a588f45948039fcd27abc9cf4 on tablet 5666124a588f45948039fcd27abc9cf4 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:06:58.012233 5965 tablet_copy_client.cc:316] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:06:58.014330 4832 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet b5088de56cbc4281b755364a8fddd084 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:06:58.015163 4832 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet b5088de56cbc4281b755364a8fddd084 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-b5088de56cbc4281b755364a8fddd084
I20260806 04:06:58.033831 4832 tablet_copy_source_session.cc:205] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:06:58.039647 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5088de56cbc4281b755364a8fddd084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:06:58.055188 5952 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:06:58.056548 5952 tablet_bootstrap.cc:492] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:06:58.057883 5952 ts_tablet_manager.cc:1397] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.023s
I20260806 04:06:58.063066 5952 raft_consensus.cc:348] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:58.065889 5952 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:06:58.066920 5952 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:58.070673 5952 ts_tablet_manager.cc:1428] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 04:06:58.073082 5965 tablet_copy_client.cc:791] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:06:58.074525 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-98064c6d64f74659bfa102bd5a341635 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:06:58.075295 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-98064c6d64f74659bfa102bd5a341635 on tablet 98064c6d64f74659bfa102bd5a341635 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:06:58.074777 5965 tablet_copy_client.cc:655] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:06:58.082072 5965 tablet_copy_client.cc:531] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:06:58.094290 5965 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:06:58.122779 5867 raft_consensus.cc:1203] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:06:58.150142 5965 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:06:58.151081 5965 tablet_bootstrap.cc:492] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:06:58.152846 5965 ts_tablet_manager.cc:1397] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.058s user 0.020s sys 0.019s
I20260806 04:06:58.155272 5965 raft_consensus.cc:348] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:58.156366 5965 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:06:58.157239 5965 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:06:58.161019 5965 ts_tablet_manager.cc:1428] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:06:58.163241 4832 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-b5088de56cbc4281b755364a8fddd084 received from {username='slave'} at 127.0.0.1:53354
I20260806 04:06:58.163646 4832 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-b5088de56cbc4281b755364a8fddd084 on tablet b5088de56cbc4281b755364a8fddd084 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:06:58.268755 5867 raft_consensus.cc:1203] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:06:58.341101 5867 raft_consensus.cc:1203] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:06:58.406302 5867 raft_consensus.cc:1203] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:06:58.719211 5713 raft_consensus.cc:1050] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:06:58.723305 5713 consensus_queue.cc:237] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:58.742136 4924 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.743073 5867 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.743798 5702 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:58.745390 5775 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 04:06:58.784746 4814 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.790681 5702 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:58.793426 5775 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.795689 5867 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.816674 4672 catalog_manager.cc:5495] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.828882 4817 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.829835 4924 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.865710 5692 raft_consensus.cc:1050] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:06:58.868430 5692 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:58.883294 5867 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.885763 4924 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.885380 5936 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:58.885907 5034 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.889170 5936 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:58.896255 5973 raft_consensus.cc:1050] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:06:58.896983 5785 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:06:58.918185 5985 raft_consensus.cc:1050] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:06:58.916603 5973 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:58.925612 5985 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:58.938642 4928 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.938802 5867 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:58.940595 5702 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:58.951426 5034 consensus_queue.cc:237] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:58.955139 5713 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:58.960888 5866 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:58.963963 5775 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:58.955761 5936 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.971452 4930 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:58.978242 5775 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:58.980841 5713 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.984299 5863 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:58.995734 4814 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:59.023540 5973 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:06:59.049109 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 72205.6us
I20260806 04:06:59.055796 4924 raft_consensus.cc:2932] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.067809 5034 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.097033 4928 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.090951 5866 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.124735 4674 catalog_manager.cc:5495] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:06:59.135684 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5666124a588f45948039fcd27abc9cf4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:06:59.164382 4817 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.170782 4924 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.157832 5973 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.185348 5867 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:59.192153 5936 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:06:59.192711 4927 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:59.160113 4676 catalog_manager.cc:5495] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.179157 4813 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:59.204277 5936 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260806 04:06:59.215579 5030 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:06:59.240674 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 5666124a588f45948039fcd27abc9cf4 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
W20260806 04:06:59.249212 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 50879.5us
I20260806 04:06:59.265707 5032 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:59.256328 5863 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.263248 6005 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:06:59.277534 4674 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.300410 5936 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
W20260806 04:06:59.305279 6004 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260806 04:06:59.316701 5867 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:59.315258 6006 tablet_replica.cc:330] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:06:59.324536 6001 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:59.356709 5692 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.359112 5867 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.360751 5985 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.366914 5034 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:59.378525 5863 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:59.383822 5998 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 04:06:59.392061 5863 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.400604 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4219680d51af4d0587436328294ba316 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:06:59.404202 4927 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:59.376564 4924 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.399320 5999 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.417979 5026 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.420663 4672 catalog_manager.cc:5495] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.418344 5867 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.438226 6006 raft_consensus.cc:2229] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:59.440117 6006 raft_consensus.cc:2258] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:59.453928 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 4219680d51af4d0587436328294ba316 with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:06:59.457455 4672 catalog_manager.cc:5495] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.461189 5713 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:59.490540 6006 ts_tablet_manager.cc:1905] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:06:59.495961 4924 raft_consensus.cc:2932] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:06:59.499004 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 46572.1us
I20260806 04:06:59.501785 4817 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:06:59.518167 5863 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:59.520360 5936 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:06:59.526087 4927 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:06:59.538695 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 98064c6d64f74659bfa102bd5a341635 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:06:59.538166 6009 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 04:06:59.536926 4674 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.544668 6010 tablet_replica.cc:330] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:06:59.563388 5026 raft_consensus.cc:2932] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.571098 5936 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.586023 4924 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.565943 5863 raft_consensus.cc:2932] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.608753 6010 raft_consensus.cc:2229] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:59.615402 6010 raft_consensus.cc:2258] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:59.621131 4672 catalog_manager.cc:5495] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:06:59.622676 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 98064c6d64f74659bfa102bd5a341635 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
I20260806 04:06:59.629272 6010 ts_tablet_manager.cc:1905] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:06:59.669999 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b5088de56cbc4281b755364a8fddd084 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:06:59.672197 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet b5088de56cbc4281b755364a8fddd084 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:06:59.697947 6006 ts_tablet_manager.cc:1918] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:06:59.698613 6006 log.cc:1196] T 5666124a588f45948039fcd27abc9cf4 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/5666124a588f45948039fcd27abc9cf4
I20260806 04:06:59.702589 6013 tablet_replica.cc:330] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:06:59.705168 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 5666124a588f45948039fcd27abc9cf4 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:06:59.712121 6006 tablet_replica.cc:330] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:06:59.713652 6013 raft_consensus.cc:2229] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:59.714963 6013 raft_consensus.cc:2258] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:59.724740 6013 ts_tablet_manager.cc:1905] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:06:59.725142 6010 ts_tablet_manager.cc:1918] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:06:59.725735 6010 log.cc:1196] T 4219680d51af4d0587436328294ba316 P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/4219680d51af4d0587436328294ba316
I20260806 04:06:59.728765 6006 raft_consensus.cc:2229] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:06:59.730763 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 4219680d51af4d0587436328294ba316 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:06:59.730068 6006 raft_consensus.cc:2258] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:06:59.738879 6006 ts_tablet_manager.cc:1905] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:06:59.749559 6013 ts_tablet_manager.cc:1918] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:06:59.750156 6013 log.cc:1196] T b5088de56cbc4281b755364a8fddd084 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/b5088de56cbc4281b755364a8fddd084
I20260806 04:06:59.752295 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet b5088de56cbc4281b755364a8fddd084 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:06:59.761400 6006 ts_tablet_manager.cc:1918] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:06:59.762048 6006 log.cc:1196] T 98064c6d64f74659bfa102bd5a341635 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/98064c6d64f74659bfa102bd5a341635
I20260806 04:06:59.768560 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 98064c6d64f74659bfa102bd5a341635 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:00.766618 4817 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:00.773366 5034 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.773447 4928 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:07:00.774117 4770 consensus_peers.cc:487] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21b536e5a86f4281a4d9ee87a11d409f. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:00.775946 5936 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:00.777413 6005 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:00.786589 6005 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.789048 5026 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.799578 4928 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.805238 4672 catalog_manager.cc:5495] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.809103 4817 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:00.826565 4770 consensus_peers.cc:487] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bf3e28a45b44b0e9c9f9f7ef333c8f5. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:00.830101 4928 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.831748 6001 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:00.832182 5026 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.833475 5936 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:00.848798 6024 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.854348 4928 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.864104 5034 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.883100 4676 catalog_manager.cc:5495] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.906143 4928 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:00.914018 5034 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.915573 6042 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:00.919049 4817 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.920459 6042 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:00.933259 6040 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:07:00.934556 4879 consensus_peers.cc:487] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12fab895c8c34b9d82a95dc9f77980c6. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:00.937706 5034 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.938028 4817 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.948760 5026 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:00.958117 4995 consensus_peers.cc:487] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5854d52dcd124250a917b57b14dd2afc. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:00.958189 4927 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.959837 5998 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:00.960481 4817 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:00.962397 4676 catalog_manager.cc:5495] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:00.965016 5998 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:00.972829 5998 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.977782 4927 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:00.982336 4809 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:01.003803 4672 catalog_manager.cc:5495] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:01.334285 6058 ts_tablet_manager.cc:927] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:01.336093 6058 tablet_copy_client.cc:316] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:01.337918 4832 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:01.338388 4832 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-6bf3e28a45b44b0e9c9f9f7ef333c8f5
I20260806 04:07:01.345100 4832 tablet_copy_source_session.cc:205] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:01.347992 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:01.362529 6058 tablet_copy_client.cc:791] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:01.363168 6058 tablet_copy_client.cc:655] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:01.367215 6058 tablet_copy_client.cc:531] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:01.375716 6058 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:01.404225 6058 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:01.407635 6060 ts_tablet_manager.cc:927] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:01.409977 6060 tablet_copy_client.cc:316] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:01.410280 6058 tablet_bootstrap.cc:492] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:01.411131 6058 ts_tablet_manager.cc:1397] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.004s
I20260806 04:07:01.412657 4832 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 21b536e5a86f4281a4d9ee87a11d409f from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:01.413225 4832 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 21b536e5a86f4281a4d9ee87a11d409f from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-21b536e5a86f4281a4d9ee87a11d409f
I20260806 04:07:01.414045 6058 raft_consensus.cc:348] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.416101 6058 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:01.416649 6058 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.421324 6058 ts_tablet_manager.cc:1428] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 04:07:01.423211 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-6bf3e28a45b44b0e9c9f9f7ef333c8f5 received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:01.423621 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-6bf3e28a45b44b0e9c9f9f7ef333c8f5 on tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:01.424526 4832 tablet_copy_source_session.cc:205] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:01.433517 6060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b536e5a86f4281a4d9ee87a11d409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:01.449453 6060 tablet_copy_client.cc:791] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:01.450116 6060 tablet_copy_client.cc:655] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:01.454042 6060 tablet_copy_client.cc:531] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:01.463353 6060 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:01.464998 6058 ts_tablet_manager.cc:927] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:01.490808 6058 tablet_copy_client.cc:316] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:01.500161 6060 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:01.501466 6060 tablet_bootstrap.cc:492] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:01.502697 6060 ts_tablet_manager.cc:1397] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.021s
I20260806 04:07:01.506614 6060 raft_consensus.cc:348] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.507920 6060 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:01.508731 6060 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.511296 6060 ts_tablet_manager.cc:1428] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260806 04:07:01.513389 4832 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-21b536e5a86f4281a4d9ee87a11d409f received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:01.513783 4832 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-21b536e5a86f4281a4d9ee87a11d409f on tablet 21b536e5a86f4281a4d9ee87a11d409f with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:01.523900 4945 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet 12fab895c8c34b9d82a95dc9f77980c6 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:01.524474 4945 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet 12fab895c8c34b9d82a95dc9f77980c6 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-12fab895c8c34b9d82a95dc9f77980c6
I20260806 04:07:01.532830 4945 tablet_copy_source_session.cc:205] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:01.536427 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12fab895c8c34b9d82a95dc9f77980c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:01.550417 6058 tablet_copy_client.cc:791] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:01.551056 6058 tablet_copy_client.cc:655] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:01.554783 6058 tablet_copy_client.cc:531] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:01.563531 6058 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:01.572321 6060 ts_tablet_manager.cc:927] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:01.579044 6060 tablet_copy_client.cc:316] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:01.581107 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 5854d52dcd124250a917b57b14dd2afc from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:01.581634 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 5854d52dcd124250a917b57b14dd2afc from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-5854d52dcd124250a917b57b14dd2afc
I20260806 04:07:01.601440 5049 tablet_copy_source_session.cc:205] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:01.610188 6060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5854d52dcd124250a917b57b14dd2afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:01.614379 6058 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:01.615309 6058 tablet_bootstrap.cc:492] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:01.616043 6058 ts_tablet_manager.cc:1397] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.030s
I20260806 04:07:01.620719 6058 raft_consensus.cc:348] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.621454 6058 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:01.621991 6058 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.627453 6060 tablet_copy_client.cc:791] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:01.628084 6060 tablet_copy_client.cc:655] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:01.629143 6058 ts_tablet_manager.cc:1428] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260806 04:07:01.631713 4945 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-12fab895c8c34b9d82a95dc9f77980c6 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:01.632153 4945 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-12fab895c8c34b9d82a95dc9f77980c6 on tablet 12fab895c8c34b9d82a95dc9f77980c6 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:01.634382 6060 tablet_copy_client.cc:531] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:01.645699 6060 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:01.667616 6060 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:01.668489 6060 tablet_bootstrap.cc:492] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:01.669185 6060 ts_tablet_manager.cc:1397] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20260806 04:07:01.673367 6060 raft_consensus.cc:348] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.674186 6060 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:01.674878 6060 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:01.678776 6060 ts_tablet_manager.cc:1428] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260806 04:07:01.680323 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-5854d52dcd124250a917b57b14dd2afc received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:01.680685 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-5854d52dcd124250a917b57b14dd2afc on tablet 5854d52dcd124250a917b57b14dd2afc with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:01.826954 5863 raft_consensus.cc:1203] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:01.884390 5863 raft_consensus.cc:1203] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:01.994787 5867 raft_consensus.cc:1203] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:01.999329 5863 raft_consensus.cc:1203] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:02.281639 5936 raft_consensus.cc:1050] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:02.286372 5936 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.297443 5863 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.300338 6081 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:07:02.305574 4924 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.307232 6081 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.317339 5030 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.319890 6081 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.334370 6081 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.337332 4928 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.352181 5863 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.367570 4672 catalog_manager.cc:5495] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:02.359002 5026 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.418833 6087 raft_consensus.cc:1050] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:02.421841 6087 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.435199 5863 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.437526 4818 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.439468 5936 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.440375 5034 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.446055 6079 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.449416 5863 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:02.451015 6081 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260806 04:07:02.462754 6081 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.464620 5863 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.476191 5034 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:02.477376 4924 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.481326 6001 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.485227 5958 raft_consensus.cc:1050] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:02.486220 4672 catalog_manager.cc:5495] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.487831 5958 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.488812 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 21b536e5a86f4281a4d9ee87a11d409f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:02.501948 5867 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.503713 5026 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.505388 6082 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.506914 5958 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:02.515501 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b536e5a86f4281a4d9ee87a11d409f with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:02.517159 4818 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.515517 6001 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260806 04:07:02.524230 6087 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.528887 4924 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.535425 5030 raft_consensus.cc:2932] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.540076 5958 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:02.555362 5034 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.565096 6106 tablet_replica.cc:330] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:02.591539 5775 raft_consensus.cc:1050] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:02.595389 6106 raft_consensus.cc:2229] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:02.595188 5775 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.596284 6106 raft_consensus.cc:2258] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:02.608659 5867 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.610643 5863 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.609287 6040 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.634919 5866 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.637609 6086 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.644193 6106 ts_tablet_manager.cc:1905] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:02.652989 4927 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.653769 4676 catalog_manager.cc:5495] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:02.650833 4668 catalog_manager.cc:5495] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.661782 4817 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.655896 5998 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:02.682337 6106 ts_tablet_manager.cc:1918] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:02.683004 6106 log.cc:1196] T 21b536e5a86f4281a4d9ee87a11d409f P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/21b536e5a86f4281a4d9ee87a11d409f
I20260806 04:07:02.684664 5775 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.690196 4813 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:02.690143 4924 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.685277 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 21b536e5a86f4281a4d9ee87a11d409f (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:02.694089 5034 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.703539 5863 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.705644 6086 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:02.730115 4927 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.731163 4668 catalog_manager.cc:5495] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.745007 4817 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:02.749384 5863 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:02.750399 4818 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.756634 4817 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:02.763397 6114 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:02.765528 6114 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:02.772554 5863 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:02.774830 6087 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:02.775138 6114 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.777446 5867 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.787070 4924 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:02.788749 6087 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:02.793033 4817 raft_consensus.cc:2932] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.796792 6079 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.801153 4674 catalog_manager.cc:5495] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.813197 5867 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.815178 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet 12fab895c8c34b9d82a95dc9f77980c6 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:02.818799 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 12fab895c8c34b9d82a95dc9f77980c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:02.822701 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260806 04:07:02.823410 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 87516us
I20260806 04:07:02.820206 6120 tablet_replica.cc:330] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:02.836946 4924 raft_consensus.cc:2932] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:02.838097 6120 raft_consensus.cc:2229] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:02.850531 6120 raft_consensus.cc:2258] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:02.854986 6120 ts_tablet_manager.cc:1905] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:02.848496 4676 catalog_manager.cc:5495] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:02.896157 6120 ts_tablet_manager.cc:1918] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:02.896636 6120 log.cc:1196] T 12fab895c8c34b9d82a95dc9f77980c6 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/12fab895c8c34b9d82a95dc9f77980c6
I20260806 04:07:02.899152 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet 12fab895c8c34b9d82a95dc9f77980c6 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:02.906826 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:02.907989 6120 tablet_replica.cc:330] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:02.910444 6120 raft_consensus.cc:2229] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:02.911208 6120 raft_consensus.cc:2258] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:02.916551 6120 ts_tablet_manager.cc:1905] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:02.934453 6120 ts_tablet_manager.cc:1918] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:02.934862 6120 log.cc:1196] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/6bf3e28a45b44b0e9c9f9f7ef333c8f5
I20260806 04:07:02.936151 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet 6bf3e28a45b44b0e9c9f9f7ef333c8f5 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:03.340539 5026 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:03.345032 5867 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:03.345993 4817 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:03.346431 5775 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:03.347370 6086 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:03.354439 5998 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:03.356456 4818 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:03.369474 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5854d52dcd124250a917b57b14dd2afc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:03.364928 5867 raft_consensus.cc:2932] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:03.379904 4676 catalog_manager.cc:5495] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:03.399901 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 5854d52dcd124250a917b57b14dd2afc with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:03.420513 6126 tablet_replica.cc:330] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:03.423812 6126 raft_consensus.cc:2229] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:03.424767 6126 raft_consensus.cc:2258] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:03.427973 6126 ts_tablet_manager.cc:1905] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:03.442958 6126 ts_tablet_manager.cc:1918] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:03.443401 6126 log.cc:1196] T 5854d52dcd124250a917b57b14dd2afc P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/5854d52dcd124250a917b57b14dd2afc
I20260806 04:07:03.444689 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 5854d52dcd124250a917b57b14dd2afc (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:04.575955 5026 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:04.590886 4995 consensus_peers.cc:487] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e1ae23ce1104b3688f3f881c64ed473. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:04.592633 4924 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:04.593678 4817 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:04.595336 5998 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:04.596843 6086 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:04.636642 5998 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.638208 4817 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.642378 4926 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.662317 5026 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:04.669304 4668 catalog_manager.cc:5495] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:07:04.688246 4995 consensus_peers.cc:487] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30e5d6887268432aaf472e51f9ae0444. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:04.709944 4926 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:04.720444 4817 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:04.723459 6136 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:04.725297 5998 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:04.769157 4818 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.765178 5998 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.801152 4924 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.840020 4668 catalog_manager.cc:5495] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.874287 4818 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:04.895455 5034 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:04.895455 4924 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:07:04.896232 4770 consensus_peers.cc:487] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a80ee0f1cc7747e8b736886e63200f37. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:04.899233 6100 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:04.901628 6146 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:04.922194 4924 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.922284 6087 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.926715 5026 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:04.948508 4818 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:04.972183 4770 consensus_peers.cc:487] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27545c8363cd404189c9e93f3bd1426c. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:04.973626 5034 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:04.984431 6100 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:05.001497 4676 catalog_manager.cc:5495] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:05.045692 4924 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:05.061043 6163 ts_tablet_manager.cc:927] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:05.064810 6128 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:05.084718 6163 tablet_copy_client.cc:316] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:05.092649 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 1e1ae23ce1104b3688f3f881c64ed473 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:05.093935 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 1e1ae23ce1104b3688f3f881c64ed473 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-1e1ae23ce1104b3688f3f881c64ed473
I20260806 04:07:05.155298 5049 tablet_copy_source_session.cc:205] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:05.175968 6163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1ae23ce1104b3688f3f881c64ed473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:05.180004 6146 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:05.200192 4926 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:05.209478 5026 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:05.238955 6163 tablet_copy_client.cc:791] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:05.240397 6163 tablet_copy_client.cc:655] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:05.251710 6163 tablet_copy_client.cc:531] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:05.250684 4672 catalog_manager.cc:5495] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:05.320703 6163 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:05.323778 6168 ts_tablet_manager.cc:927] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:05.346223 6168 tablet_copy_client.cc:316] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:05.353953 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 30e5d6887268432aaf472e51f9ae0444 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:05.355187 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 30e5d6887268432aaf472e51f9ae0444 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-30e5d6887268432aaf472e51f9ae0444
I20260806 04:07:05.394927 5049 tablet_copy_source_session.cc:205] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:05.403793 6168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e5d6887268432aaf472e51f9ae0444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:05.494944 6175 ts_tablet_manager.cc:927] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:05.497258 6172 ts_tablet_manager.cc:927] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:05.565804 6172 tablet_copy_client.cc:316] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:05.570747 6168 tablet_copy_client.cc:791] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:05.572396 6168 tablet_copy_client.cc:655] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:05.590708 6168 tablet_copy_client.cc:531] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:05.600755 4832 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 27545c8363cd404189c9e93f3bd1426c from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:05.602028 4832 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 27545c8363cd404189c9e93f3bd1426c from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-27545c8363cd404189c9e93f3bd1426c
I20260806 04:07:05.608371 6175 tablet_copy_client.cc:316] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:05.611338 6163 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:05.627774 4831 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet a80ee0f1cc7747e8b736886e63200f37 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:05.628247 6168 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:05.631919 4831 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet a80ee0f1cc7747e8b736886e63200f37 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-a80ee0f1cc7747e8b736886e63200f37
I20260806 04:07:05.630342 4832 tablet_copy_source_session.cc:205] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:05.635497 6163 tablet_bootstrap.cc:492] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:05.636744 6163 ts_tablet_manager.cc:1397] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.317s user 0.068s sys 0.059s
I20260806 04:07:05.642131 6172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27545c8363cd404189c9e93f3bd1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:05.672981 4831 tablet_copy_source_session.cc:205] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:05.677992 6163 raft_consensus.cc:348] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:05.710999 6163 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:05.715085 6163 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:05.722134 6172 tablet_copy_client.cc:791] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:05.723775 6172 tablet_copy_client.cc:655] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:05.761370 6163 ts_tablet_manager.cc:1428] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.123s user 0.018s sys 0.003s
I20260806 04:07:05.787684 6168 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:05.789094 6168 tablet_bootstrap.cc:492] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:05.789901 6168 ts_tablet_manager.cc:1397] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.162s user 0.047s sys 0.037s
I20260806 04:07:05.798944 6168 raft_consensus.cc:348] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:05.801620 6168 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:05.804286 6168 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:05.812532 6168 ts_tablet_manager.cc:1428] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.022s user 0.017s sys 0.005s
I20260806 04:07:05.876542 5048 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-30e5d6887268432aaf472e51f9ae0444 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:05.877132 5048 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-30e5d6887268432aaf472e51f9ae0444 on tablet 30e5d6887268432aaf472e51f9ae0444 with peer d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:05.912516 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 256108us
I20260806 04:07:05.929044 6175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80ee0f1cc7747e8b736886e63200f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:05.952152 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-1e1ae23ce1104b3688f3f881c64ed473 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:05.953065 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-1e1ae23ce1104b3688f3f881c64ed473 on tablet 1e1ae23ce1104b3688f3f881c64ed473 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:05.985951 6172 tablet_copy_client.cc:531] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:06.016577 6175 tablet_copy_client.cc:791] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
W20260806 04:07:06.151610 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 83533.4us
W20260806 04:07:06.189260 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 94114.8us
I20260806 04:07:06.368366 5867 raft_consensus.cc:1203] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:06.489257 5867 raft_consensus.cc:1203] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:06.568267 6175 tablet_copy_client.cc:655] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
W20260806 04:07:06.621851 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 251296us
W20260806 04:07:06.629894 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 251328us
I20260806 04:07:06.671407 6172 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:06.671593 6167 raft_consensus.cc:1050] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:06.743584 5998 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.744803 5998 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:06.749914 5998 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:06.761726 6217 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.763700 6217 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:06.778640 6217 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:06.789407 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46412us
I20260806 04:07:06.796415 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:06.797293 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:06.815222 6183 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.816440 6136 raft_consensus.cc:1050] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:06.816475 6183 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:06.785917 6167 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:06.826623 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 136921us
I20260806 04:07:06.832715 6183 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:06.849673 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 57445.7us
I20260806 04:07:06.917110 6217 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.918713 6217 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:06.922909 6194 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.924337 6194 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:06.941497 6086 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.943262 6086 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:06.952322 6215 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:06.956372 6194 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:06.959836 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:06.961956 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:06.949611 6086 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:06.985499 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:06.997910 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:07.006040 5034 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:06.953857 6215 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:07.026248 5067 debug-util.cc:398] Leaking SignalData structure 0x7b08000c78a0 after lost signal to thread 3045
I20260806 04:07:07.030032 6175 tablet_copy_client.cc:531] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260806 04:07:07.031956 5067 debug-util.cc:398] Leaking SignalData structure 0x7b08001d2c80 after lost signal to thread 4724
I20260806 04:07:07.045362 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.028518 6215 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:07.063681 5067 debug-util.cc:398] Leaking SignalData structure 0x7b080005eea0 after lost signal to thread 4853
I20260806 04:07:07.040288 6136 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:07.081470 5067 debug-util.cc:398] Leaking SignalData structure 0x7b08002b7f40 after lost signal to thread 4966
I20260806 04:07:07.008543 6086 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:07.133740 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 114028us
I20260806 04:07:07.145201 6186 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:07.172040 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 149092us
W20260806 04:07:07.185896 6223 log.cc:924] Time spent T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.073s user 0.000s sys 0.003s
I20260806 04:07:07.194804 5867 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:07.196130 4927 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:07.197149 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.142109 6086 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.202610 5998 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:07.174125 5067 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff960 after lost signal to thread 5070
I20260806 04:07:07.211452 5998 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.146528 6186 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.187479 6217 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:07.235491 6086 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:07.225416 5998 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:07.248004 5067 debug-util.cc:398] Leaking SignalData structure 0x7b08002f91a0 after lost signal to thread 5901
I20260806 04:07:07.213181 6215 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260806 04:07:07.259683 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.261521 6086 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:07.269634 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.296396 6186 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:07.321172 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.328439 6175 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:07.312690 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.342555 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.352802 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.353648 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.355611 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.367267 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.375820 5866 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:07.384784 6172 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:07.386147 6172 tablet_bootstrap.cc:492] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:07.387300 4924 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:07.387988 4996 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.357996 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.389375 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.390789 4996 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.394847 5998 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.395983 6217 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:07.396136 6196 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.402679 6184 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.402201 4996 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.414984 6172 ts_tablet_manager.cc:1397] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.744s user 0.028s sys 0.077s
I20260806 04:07:07.415709 6136 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.415273 4996 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.430043 6226 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.433233 6186 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:07:07.435932 4996 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:07.463985 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 279727us
I20260806 04:07:07.468987 4996 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.437873 6215 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.480613 4768 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:07.484992 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.493999 4805 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:07.499462 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.502506 4808 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:07.504714 6215 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.506583 6186 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:07.508224 6184 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:07.516173 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:07.489459 6217 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.534759 6229 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:07.535593 6229 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:07.536185 6229 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:07.474927 6172 raft_consensus.cc:348] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:07.538494 4996 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:07.543103 6100 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:07.539076 6172 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:07.543884 6100 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:07.548096 6100 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:07.544715 6172 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:07.546097 6215 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.552798 6199 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:07.570109 6217 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.572084 6199 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.591389 6199 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:07.609484 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.621106 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:07.621954 5863 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.601190 6199 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:07.629312 6199 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.632407 6199 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:07.701584 4924 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.709693 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.717262 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:07.719343 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:07.720240 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:07.720657 6196 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.728421 5866 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.725842 4768 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:07.739236 4813 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.752491 6229 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.762718 6229 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:07.764748 4768 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:07.773941 6199 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.778100 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:07.779237 6199 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.786190 4672 catalog_manager.cc:5495] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.789742 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:07.793848 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:07.793511 4668 catalog_manager.cc:5495] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.769847 6183 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.806103 4768 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:07.825448 4808 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.809036 6199 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:07.830999 4924 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:07.844511 4813 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:07.846050 6202 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 04:07:07.853616 6172 ts_tablet_manager.cc:1428] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.435s user 0.026s sys 0.027s
I20260806 04:07:07.861009 6220 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:07.861737 6220 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:07.862118 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-27545c8363cd404189c9e93f3bd1426c received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:07.862852 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-27545c8363cd404189c9e93f3bd1426c on tablet 27545c8363cd404189c9e93f3bd1426c with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:07.874704 5863 raft_consensus.cc:1203] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:07.863245 6220 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:07.900931 6220 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:07.931870 6175 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:07.949340 6175 tablet_bootstrap.cc:492] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:07.951447 6175 ts_tablet_manager.cc:1397] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.623s user 0.040s sys 0.024s
I20260806 04:07:07.962929 6175 raft_consensus.cc:348] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:07.964282 6175 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:07.965060 6175 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:07.974851 6175 ts_tablet_manager.cc:1428] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.023s user 0.004s sys 0.006s
I20260806 04:07:07.977309 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-a80ee0f1cc7747e8b736886e63200f37 received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:07.977927 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-a80ee0f1cc7747e8b736886e63200f37 on tablet a80ee0f1cc7747e8b736886e63200f37 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:08.021052 5034 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:08.030340 5863 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.032583 4924 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.036216 5034 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:08.038471 6167 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260806 04:07:08.044755 5863 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.045056 6184 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:08.046394 6167 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:08.048996 4924 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.055742 6196 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.057376 6217 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:08.060005 4924 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.065466 5863 raft_consensus.cc:2932] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.071144 6186 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.081629 4927 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.081910 5866 raft_consensus.cc:2932] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.109244 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1e1ae23ce1104b3688f3f881c64ed473 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:08.148535 4668 catalog_manager.cc:5495] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.153604 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 30e5d6887268432aaf472e51f9ae0444 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:08.168742 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1ae23ce1104b3688f3f881c64ed473 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
I20260806 04:07:08.180612 6248 tablet_replica.cc:330] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:07:08.182420 6248 raft_consensus.cc:2229] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:08.183429 6248 raft_consensus.cc:2258] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:08.187019 4672 catalog_manager.cc:5495] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:08.193383 6248 ts_tablet_manager.cc:1905] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:08.203163 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e5d6887268432aaf472e51f9ae0444 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
I20260806 04:07:08.315107 6248 ts_tablet_manager.cc:1918] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:08.315635 6248 log.cc:1196] T 1e1ae23ce1104b3688f3f881c64ed473 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/1e1ae23ce1104b3688f3f881c64ed473
I20260806 04:07:08.318552 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 1e1ae23ce1104b3688f3f881c64ed473 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:08.324919 6248 tablet_replica.cc:330] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:07:08.339943 6248 raft_consensus.cc:2229] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:08.341591 6248 raft_consensus.cc:2258] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:08.345911 6248 ts_tablet_manager.cc:1905] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:08.356172 5866 raft_consensus.cc:1203] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:08.398330 6248 ts_tablet_manager.cc:1918] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:08.410676 6248 log.cc:1196] T 30e5d6887268432aaf472e51f9ae0444 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/30e5d6887268432aaf472e51f9ae0444
I20260806 04:07:08.412739 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 30e5d6887268432aaf472e51f9ae0444 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:08.482460 6199 raft_consensus.cc:1050] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:08.491144 6199 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:08.509195 5034 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:08.509359 5866 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:08.510890 6199 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:08.511801 4924 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:08.511974 6183 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:08.517026 6199 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:08.528854 6199 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.539358 4928 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.532595 5867 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.553130 5032 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.565907 4676 catalog_manager.cc:5495] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.609743 4816 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:08.619335 5867 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.622485 6100 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:08.629570 6199 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.630913 5863 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.636540 5032 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.640775 6100 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:08.645476 5032 raft_consensus.cc:2932] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.701253 4676 catalog_manager.cc:5495] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:08.707557 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 53627.2us
I20260806 04:07:08.714596 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 27545c8363cd404189c9e93f3bd1426c with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:08.729135 6254 tablet_replica.cc:330] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:08.730171 6254 raft_consensus.cc:2229] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:08.730966 6254 raft_consensus.cc:2258] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:08.732113 6204 raft_consensus.cc:1050] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:08.737107 6254 ts_tablet_manager.cc:1905] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:08.749752 6204 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:08.755676 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 27545c8363cd404189c9e93f3bd1426c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:08.756131 6254 ts_tablet_manager.cc:1918] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:08.756558 6254 log.cc:1196] T 27545c8363cd404189c9e93f3bd1426c P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/27545c8363cd404189c9e93f3bd1426c
I20260806 04:07:08.758369 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 27545c8363cd404189c9e93f3bd1426c (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:08.769805 5863 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:08.773775 5032 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:08.775305 4924 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:08.776613 6219 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:08.779330 6229 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:08.780242 6199 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:08.805032 6204 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.809998 5863 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.812314 4924 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.828401 5034 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.840683 4676 catalog_manager.cc:5495] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.860808 4816 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:08.884461 5863 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.885965 6199 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:08.892925 6229 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.895387 4924 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:08.894599 5867 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.897289 6100 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:08.912887 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a80ee0f1cc7747e8b736886e63200f37 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:08.911962 4924 raft_consensus.cc:2932] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.916092 4674 catalog_manager.cc:5495] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:08.928615 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet a80ee0f1cc7747e8b736886e63200f37 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:08.931615 6262 tablet_replica.cc:330] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:08.932695 6262 raft_consensus.cc:2229] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:08.933454 6262 raft_consensus.cc:2258] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:08.941231 6262 ts_tablet_manager.cc:1905] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:08.959826 6262 ts_tablet_manager.cc:1918] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:08.960256 6262 log.cc:1196] T a80ee0f1cc7747e8b736886e63200f37 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/a80ee0f1cc7747e8b736886e63200f37
I20260806 04:07:08.961916 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet a80ee0f1cc7747e8b736886e63200f37 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:09.053107 6136 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:09.053704 6136 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:09.055565 6136 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:09.056461 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:09.056658 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:09.057466 4924 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:09.057824 4808 raft_consensus.cc:2379] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:09.058477 4992 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:09.059286 6136 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:09.059666 6136 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:09.060235 6136 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:09.070245 6136 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:09.072810 6136 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:09.073668 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:09.074779 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:09.074769 4808 raft_consensus.cc:2379] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:09.080209 4924 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:09.081167 4992 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:09.082017 6196 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:09.082693 6196 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:09.083834 6196 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:09.111233 4672 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:09.477025 4808 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:09.478422 6136 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:09.510936 4926 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:09.512867 6196 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:10.006603 4816 consensus_queue.cc:237] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:10.032445 4770 consensus_peers.cc:487] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 589278a46368433c87a150021544ba2e. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:10.032696 5034 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:10.034173 6199 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:10.035620 4924 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:10.040130 6199 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:10.095693 6199 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.099695 4927 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.123121 4672 catalog_manager.cc:5495] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:10.113529 5032 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.167914 5035 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:07:10.184598 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 173750us
W20260806 04:07:10.220309 4995 consensus_peers.cc:487] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78dcfac52d99420a82f5c7b402a782d7. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:10.220343 4816 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:07:10.221700 4928 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:07:10.230513 6272 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:10.237939 6136 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.250524 4816 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.261983 6272 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:10.272500 4816 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.273293 4676 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.285308 4928 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.307344 4927 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:10.308837 6278 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:10.311225 5032 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:07:10.312758 4770 consensus_peers.cc:487] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa361eb5ab5463895c3f7af5e87237c. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:10.314987 6100 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:10.319617 6199 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.327462 4926 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.351423 5030 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.367487 4676 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.422605 5032 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.494856 6301 ts_tablet_manager.cc:927] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
W20260806 04:07:10.505646 4995 consensus_peers.cc:487] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7659c18c5db84b729be8dce3bcb9dd32. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:10.508723 6301 tablet_copy_client.cc:316] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:10.515173 4831 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 589278a46368433c87a150021544ba2e from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:10.515828 4831 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 589278a46368433c87a150021544ba2e from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-589278a46368433c87a150021544ba2e
I20260806 04:07:10.521039 4816 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:10.523351 4928 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:10.524490 6136 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:10.540570 6280 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.542886 4813 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.568969 4674 catalog_manager.cc:5495] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.572587 4831 tablet_copy_source_session.cc:205] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:10.578739 6196 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:10.585530 6301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 589278a46368433c87a150021544ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:10.646876 6301 tablet_copy_client.cc:791] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:10.647650 6301 tablet_copy_client.cc:655] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:10.663949 6301 tablet_copy_client.cc:531] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:10.681021 6305 ts_tablet_manager.cc:927] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:10.673014 4928 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:10.687350 6301 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:10.700639 6305 tablet_copy_client.cc:316] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:10.703083 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 78dcfac52d99420a82f5c7b402a782d7 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:10.703672 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 78dcfac52d99420a82f5c7b402a782d7 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-78dcfac52d99420a82f5c7b402a782d7
I20260806 04:07:10.713078 5049 tablet_copy_source_session.cc:205] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:10.718000 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dcfac52d99420a82f5c7b402a782d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:10.743645 6305 tablet_copy_client.cc:791] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:10.744728 6305 tablet_copy_client.cc:655] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:10.754298 6305 tablet_copy_client.cc:531] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:10.776358 6305 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:10.777086 6301 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:10.778424 6301 tablet_bootstrap.cc:492] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:10.779485 6301 ts_tablet_manager.cc:1397] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.092s user 0.030s sys 0.007s
I20260806 04:07:10.786479 6301 raft_consensus.cc:348] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.787747 6301 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:10.788838 6301 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.848548 6301 ts_tablet_manager.cc:1428] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.068s user 0.020s sys 0.017s
I20260806 04:07:10.879390 6305 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:10.881207 6305 tablet_bootstrap.cc:492] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:10.882488 6305 ts_tablet_manager.cc:1397] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.041s
I20260806 04:07:10.895860 6305 raft_consensus.cc:348] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.912964 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-589278a46368433c87a150021544ba2e received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:10.916811 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-589278a46368433c87a150021544ba2e on tablet 589278a46368433c87a150021544ba2e with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:10.918833 6305 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:10.919970 6305 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:10.927052 6301 ts_tablet_manager.cc:927] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:10.934068 6316 ts_tablet_manager.cc:927] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:10.935866 6305 ts_tablet_manager.cc:1428] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.053s user 0.006s sys 0.003s
I20260806 04:07:10.961786 6316 tablet_copy_client.cc:316] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:10.951123 6301 tablet_copy_client.cc:316] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:10.963442 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-78dcfac52d99420a82f5c7b402a782d7 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:10.963933 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-78dcfac52d99420a82f5c7b402a782d7 on tablet 78dcfac52d99420a82f5c7b402a782d7 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:10.969064 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 7659c18c5db84b729be8dce3bcb9dd32 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:10.969770 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 7659c18c5db84b729be8dce3bcb9dd32 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-7659c18c5db84b729be8dce3bcb9dd32
I20260806 04:07:10.971498 4831 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet faa361eb5ab5463895c3f7af5e87237c from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:10.972395 4831 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet faa361eb5ab5463895c3f7af5e87237c from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-faa361eb5ab5463895c3f7af5e87237c
I20260806 04:07:10.980248 5049 tablet_copy_source_session.cc:205] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:10.985489 4831 tablet_copy_source_session.cc:205] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:10.986310 6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659c18c5db84b729be8dce3bcb9dd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:10.989053 6301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa361eb5ab5463895c3f7af5e87237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:11.019379 6301 tablet_copy_client.cc:791] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:11.021751 6301 tablet_copy_client.cc:655] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:11.016993 5867 raft_consensus.cc:1203] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:11.033411 6316 tablet_copy_client.cc:791] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:11.034060 6316 tablet_copy_client.cc:655] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:11.038947 6316 tablet_copy_client.cc:531] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:11.045217 6301 tablet_copy_client.cc:531] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:11.054409 6316 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:11.062071 6301 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:11.121917 6316 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:11.127457 6316 tablet_bootstrap.cc:492] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:11.128258 6316 ts_tablet_manager.cc:1397] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.074s user 0.018s sys 0.020s
I20260806 04:07:11.130972 6316 raft_consensus.cc:348] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:11.131906 6316 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:11.132576 6316 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:11.134428 6316 ts_tablet_manager.cc:1428] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:07:11.142199 6301 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:11.143479 6301 tablet_bootstrap.cc:492] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:11.144291 6301 ts_tablet_manager.cc:1397] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.082s user 0.031s sys 0.016s
I20260806 04:07:11.147228 6301 raft_consensus.cc:348] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:11.148173 6301 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:11.148877 6301 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:11.151335 6301 ts_tablet_manager.cc:1428] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:07:11.154474 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-7659c18c5db84b729be8dce3bcb9dd32 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:11.155303 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-7659c18c5db84b729be8dce3bcb9dd32 on tablet 7659c18c5db84b729be8dce3bcb9dd32 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:11.155458 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-faa361eb5ab5463895c3f7af5e87237c received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:11.156056 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-faa361eb5ab5463895c3f7af5e87237c on tablet faa361eb5ab5463895c3f7af5e87237c with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:11.209098 5863 raft_consensus.cc:1203] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:07:11.411873 6300 raft_consensus.cc:1050] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:11.414681 6300 consensus_queue.cc:237] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:11.419314 5867 raft_consensus.cc:1203] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:11.433122 5867 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:11.434612 6278 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:11.441625 4928 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:11.445900 6322 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:11.466121 5863 raft_consensus.cc:1203] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:11.483283 6278 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.489024 4927 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.492197 5026 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:11.493777 6322 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:11.507762 5030 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.508069 4676 catalog_manager.cc:5495] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.523639 5867 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.607125 6280 raft_consensus.cc:1050] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:11.610081 6280 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:11.618268 5867 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:07:11.619894 6196 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:11.622144 4928 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:07:11.623641 6136 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:11.630621 4810 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:07:11.634140 6196 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:11.654341 6136 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.673349 4928 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.688334 4810 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.696620 4668 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.712612 5867 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.727989 5030 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:11.772886 5867 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:11.773723 4928 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:11.775372 6345 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:11.778457 6196 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:11.783918 6280 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.788035 5867 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.796676 4927 raft_consensus.cc:2932] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.799897 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 78dcfac52d99420a82f5c7b402a782d7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:11.809314 4672 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 4 to 5, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:11.828373 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 78dcfac52d99420a82f5c7b402a782d7 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33546
I20260806 04:07:11.841642 6347 tablet_replica.cc:330] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:07:11.843004 6347 raft_consensus.cc:2229] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:11.843897 6347 raft_consensus.cc:2258] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:11.850759 6347 ts_tablet_manager.cc:1905] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:11.876546 6347 ts_tablet_manager.cc:1918] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 04:07:11.877142 6347 log.cc:1196] T 78dcfac52d99420a82f5c7b402a782d7 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/78dcfac52d99420a82f5c7b402a782d7
I20260806 04:07:11.879122 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 78dcfac52d99420a82f5c7b402a782d7 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:12.018587 6336 raft_consensus.cc:1050] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:12.020954 6336 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:12.035333 5867 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:12.035687 5030 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:12.036221 4927 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:12.038884 6300 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:12.039911 6336 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:12.040971 6337 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:12.050218 6345 raft_consensus.cc:1050] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:12.053926 6345 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:12.115789 4810 consensus_queue.cc:237] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:12.123780 5867 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:12.127969 5030 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:12.130299 6336 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260806 04:07:12.150985 5867 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:12.162544 6136 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:12.171972 4927 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:12.172523 4810 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:12.174686 6300 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.180774 6196 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:12.180912 6337 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:12.187120 6136 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:07:12.232106 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 50053.5us
I20260806 04:07:12.235585 5863 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:12.238868 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 57658.2us
I20260806 04:07:12.244284 6278 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.252617 6136 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.260905 4927 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.279634 5867 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:12.319758 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 71627.3us
I20260806 04:07:12.339973 4926 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.357568 4668 catalog_manager.cc:5495] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:12.369680 5030 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.371928 4672 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.390928 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 589278a46368433c87a150021544ba2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:12.391347 4816 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.361824 5863 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.517174 5026 raft_consensus.cc:2932] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.581506 4816 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:12.602579 5867 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:12.604305 6300 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:12.606930 4928 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:12.586747 4674 catalog_manager.cc:5495] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:12.614293 6359 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.620941 6363 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:12.623500 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 589278a46368433c87a150021544ba2e with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:12.626533 6382 tablet_replica.cc:330] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:12.628495 6382 raft_consensus.cc:2229] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:12.629750 6382 raft_consensus.cc:2258] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:12.631088 5026 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:12.633270 6382 ts_tablet_manager.cc:1905] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:12.632829 5863 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.638170 4928 raft_consensus.cc:2932] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.639263 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet faa361eb5ab5463895c3f7af5e87237c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:12.663173 4674 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:12.677551 5863 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:12.679215 6136 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:12.686599 6345 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.688076 5867 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.693969 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet faa361eb5ab5463895c3f7af5e87237c with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:12.718906 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7659c18c5db84b729be8dce3bcb9dd32 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:12.722596 4928 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:12.724306 6136 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:12.728595 4926 raft_consensus.cc:2932] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:12.740459 4676 catalog_manager.cc:5495] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:12.744216 6382 ts_tablet_manager.cc:1918] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:12.745684 6382 log.cc:1196] T 589278a46368433c87a150021544ba2e P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/589278a46368433c87a150021544ba2e
I20260806 04:07:12.741607 6383 tablet_replica.cc:330] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:12.752385 6383 raft_consensus.cc:2229] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:12.755365 6383 raft_consensus.cc:2258] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:12.760238 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 589278a46368433c87a150021544ba2e (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:12.761366 6383 ts_tablet_manager.cc:1905] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:12.771224 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 7659c18c5db84b729be8dce3bcb9dd32 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
I20260806 04:07:12.781900 6383 ts_tablet_manager.cc:1918] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:12.782452 6383 log.cc:1196] T faa361eb5ab5463895c3f7af5e87237c P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/faa361eb5ab5463895c3f7af5e87237c
I20260806 04:07:12.788609 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet faa361eb5ab5463895c3f7af5e87237c (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:12.816478 6384 tablet_replica.cc:330] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:07:12.817642 6384 raft_consensus.cc:2229] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:12.818488 6384 raft_consensus.cc:2258] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:12.825304 6384 ts_tablet_manager.cc:1905] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:12.851835 6384 ts_tablet_manager.cc:1918] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:12.852421 6384 log.cc:1196] T 7659c18c5db84b729be8dce3bcb9dd32 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/7659c18c5db84b729be8dce3bcb9dd32
I20260806 04:07:12.853886 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 7659c18c5db84b729be8dce3bcb9dd32 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:13.829326 4928 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:13.842231 5026 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:13.843564 4813 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:13.843892 6271 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:07:13.845105 4879 consensus_peers.cc:487] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f75362215d584713a892bd7319a22469. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:13.845691 6271 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:13.854272 6271 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:13.858021 4813 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:13.862064 5030 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:13.885550 4674 catalog_manager.cc:5495] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:13.892686 4813 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:13.924052 4770 consensus_peers.cc:487] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cb317a04be74515913f7aee753c193b. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:13.937896 5026 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:13.940555 6385 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:13.941994 4928 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:13.946455 6359 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:13.991385 6385 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:13.994686 4926 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.002776 5026 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.023793 4928 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:14.024606 4676 catalog_manager.cc:5495] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.038578 5026 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:14.039798 4813 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:14.040258 6386 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:14.042132 6386 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:07:14.061780 4879 consensus_peers.cc:487] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3ae3b855e0d4f98a67493c708f71d33. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:14.069792 6271 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.071373 4813 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.091298 4926 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:14.095847 4668 catalog_manager.cc:5495] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.098078 5026 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:07:14.121800 4879 consensus_peers.cc:487] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 793f3bf4a49a4d108609efd686fb6c85. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:14.123853 5032 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:14.125530 6386 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:14.136117 4813 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:14.137751 6271 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:14.141847 6386 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.149474 5032 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.192174 4813 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:14.186439 4668 catalog_manager.cc:5495] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:14.438083 6416 ts_tablet_manager.cc:927] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:14.642445 6425 ts_tablet_manager.cc:927] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:14.688503 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 240909us
W20260806 04:07:14.692116 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 283051us
I20260806 04:07:14.712620 6416 tablet_copy_client.cc:316] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:14.871484 6433 ts_tablet_manager.cc:927] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
W20260806 04:07:14.872640 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 165523us
I20260806 04:07:14.822198 6425 tablet_copy_client.cc:316] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:15.144905 6441 ts_tablet_manager.cc:927] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:15.181710 6433 tablet_copy_client.cc:316] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:15.290618 4944 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet a3ae3b855e0d4f98a67493c708f71d33 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:15.292641 4944 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet a3ae3b855e0d4f98a67493c708f71d33 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-a3ae3b855e0d4f98a67493c708f71d33
I20260806 04:07:15.293361 4945 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet f75362215d584713a892bd7319a22469 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:15.294262 4945 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet f75362215d584713a892bd7319a22469 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-f75362215d584713a892bd7319a22469
I20260806 04:07:15.308151 4831 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 7cb317a04be74515913f7aee753c193b from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:15.308877 4831 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 7cb317a04be74515913f7aee753c193b from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-7cb317a04be74515913f7aee753c193b
W20260806 04:07:15.326452 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 53475.9us
I20260806 04:07:15.358224 4944 tablet_copy_source_session.cc:205] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
W20260806 04:07:15.392545 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 46659.3us
I20260806 04:07:15.393883 6436 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:15.395157 6436 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:15.397519 6436 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:15.364023 4831 tablet_copy_source_session.cc:205] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:15.339509 6441 tablet_copy_client.cc:316] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:15.404656 4943 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet 793f3bf4a49a4d108609efd686fb6c85 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
W20260806 04:07:15.404708 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 87706.2us
I20260806 04:07:15.405433 4943 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet 793f3bf4a49a4d108609efd686fb6c85 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-793f3bf4a49a4d108609efd686fb6c85
I20260806 04:07:15.407500 4945 tablet_copy_source_session.cc:205] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:15.418933 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75362215d584713a892bd7319a22469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:15.432751 4943 tablet_copy_source_session.cc:205] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:15.438712 6441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f3bf4a49a4d108609efd686fb6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:15.442561 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ae3b855e0d4f98a67493c708f71d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:15.442201 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:15.446805 6416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45574 (request call id 35) took 1046 ms. Trace:
I20260806 04:07:15.447283 6416 rpcz_store.cc:276] 0806 04:07:14.400539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:14.402794 (+ 2255us) service_pool.cc:227] Handling call
0806 04:07:14.502251 (+ 99457us) ts_tablet_manager.cc:928] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
0806 04:07:15.417391 (+915140us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:07:15.433008 (+ 15617us) tablet_metadata.cc:641] Metadata flushed
0806 04:07:15.433323 (+ 315us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:07:15.444951 (+ 11628us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:07:15.446206 (+ 1255us) tablet_replica.cc:205] Creating consensus instance
0806 04:07:15.446614 (+ 408us) ts_tablet_manager.cc:983] Replica registered.
0806 04:07:15.446718 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":14444,"tablet-copy.queue_time_us":32420,"thread_start_us":88268,"threads_started":2}
I20260806 04:07:15.449193 6416 tablet_copy_client.cc:791] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:15.449846 6416 tablet_copy_client.cc:655] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:15.451897 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:15.453658 6433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb317a04be74515913f7aee753c193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:15.456911 6416 tablet_copy_client.cc:531] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:15.480480 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:15.483067 6457 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:15.505688 6441 tablet_copy_client.cc:791] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:15.506474 6441 tablet_copy_client.cc:655] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:15.510780 6441 tablet_copy_client.cc:531] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:15.520464 6441 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:15.542325 6425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45574 (request call id 36) took 1086 ms. Trace:
I20260806 04:07:15.549137 6425 rpcz_store.cc:276] 0806 04:07:14.491739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:14.520138 (+ 28399us) service_pool.cc:227] Handling call
0806 04:07:14.643989 (+123851us) ts_tablet_manager.cc:928] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
0806 04:07:15.441195 (+797206us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:07:15.493145 (+ 51950us) tablet_metadata.cc:641] Metadata flushed
0806 04:07:15.493436 (+ 291us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:07:15.541163 (+ 47727us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:07:15.541798 (+ 635us) tablet_replica.cc:205] Creating consensus instance
0806 04:07:15.542185 (+ 387us) ts_tablet_manager.cc:983] Replica registered.
0806 04:07:15.542258 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":79061,"tablet-copy.queue_time_us":115383,"thread_start_us":186304,"threads_started":2}
I20260806 04:07:15.550865 6425 tablet_copy_client.cc:791] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:15.544175 6433 tablet_copy_client.cc:791] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:15.555442 6433 tablet_copy_client.cc:655] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:15.561829 6433 tablet_copy_client.cc:531] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:15.551551 6425 tablet_copy_client.cc:655] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:15.588048 6425 tablet_copy_client.cc:531] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:15.590209 6416 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:15.599673 6441 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:15.600800 6441 tablet_bootstrap.cc:492] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:15.601677 6441 ts_tablet_manager.cc:1397] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.081s user 0.040s sys 0.018s
I20260806 04:07:15.608428 6433 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:15.612378 6425 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:15.615976 6441 raft_consensus.cc:348] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.616967 6441 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:15.623427 6441 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.636919 6441 ts_tablet_manager.cc:1428] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.035s user 0.013s sys 0.000s
I20260806 04:07:15.640151 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-793f3bf4a49a4d108609efd686fb6c85 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:15.640620 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-793f3bf4a49a4d108609efd686fb6c85 on tablet 793f3bf4a49a4d108609efd686fb6c85 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:15.697057 6385 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:15.697665 6385 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.699985 6385 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:15.700899 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:15.702260 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:15.710346 6416 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:15.711512 6416 tablet_bootstrap.cc:492] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:15.712456 6416 ts_tablet_manager.cc:1397] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.126s user 0.023s sys 0.027s
I20260806 04:07:15.713981 4768 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:15.715704 6416 raft_consensus.cc:348] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.716655 6416 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:15.717060 6426 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:15.717397 6416 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.723044 6416 ts_tablet_manager.cc:1428] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260806 04:07:15.730978 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-f75362215d584713a892bd7319a22469 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:15.731437 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-f75362215d584713a892bd7319a22469 on tablet f75362215d584713a892bd7319a22469 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:15.745301 6433 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:15.752103 6433 tablet_bootstrap.cc:492] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:15.752926 6433 ts_tablet_manager.cc:1397] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.145s user 0.015s sys 0.017s
I20260806 04:07:15.766145 6433 raft_consensus.cc:348] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.767203 6433 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:15.767966 6433 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.769899 5863 raft_consensus.cc:1203] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:15.795059 6433 ts_tablet_manager.cc:1428] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.038s user 0.007s sys 0.001s
I20260806 04:07:15.796772 6425 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:15.797780 6425 tablet_bootstrap.cc:492] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:15.807212 6425 ts_tablet_manager.cc:1397] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.207s user 0.027s sys 0.042s
I20260806 04:07:15.810310 6425 raft_consensus.cc:348] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.811841 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-7cb317a04be74515913f7aee753c193b received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:15.812319 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-7cb317a04be74515913f7aee753c193b on tablet 7cb317a04be74515913f7aee753c193b with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:15.811292 6425 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:15.820978 6425 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:15.824296 6425 ts_tablet_manager.cc:1428] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20260806 04:07:15.838063 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-a3ae3b855e0d4f98a67493c708f71d33 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:15.838569 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-a3ae3b855e0d4f98a67493c708f71d33 on tablet a3ae3b855e0d4f98a67493c708f71d33 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:16.061841 5863 raft_consensus.cc:1203] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:16.137526 5863 raft_consensus.cc:1203] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:16.187299 5863 raft_consensus.cc:1203] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:16.212172 6413 raft_consensus.cc:1050] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:16.215181 6413 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:16.241099 5035 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.242628 6450 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.248227 4817 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.250152 6452 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:16.241582 5863 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.256302 6452 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 04:07:16.289206 6457 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.291183 5862 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.300091 4809 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.317790 4674 catalog_manager.cc:5495] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.335439 5035 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.393723 4923 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:16.400110 5862 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:16.402698 6452 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:16.403200 5030 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:16.411527 6452 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:16.423669 6450 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.433089 5862 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.435549 5030 raft_consensus.cc:2932] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.437503 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 793f3bf4a49a4d108609efd686fb6c85 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:16.465982 4674 catalog_manager.cc:5495] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:16.494962 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 793f3bf4a49a4d108609efd686fb6c85 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
I20260806 04:07:16.501091 6488 tablet_replica.cc:330] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:07:16.513080 6488 raft_consensus.cc:2229] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:16.514149 6488 raft_consensus.cc:2258] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:16.518250 6488 ts_tablet_manager.cc:1905] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:16.543633 6488 ts_tablet_manager.cc:1918] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:16.544202 6385 raft_consensus.cc:1050] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:16.544313 6488 log.cc:1196] T 793f3bf4a49a4d108609efd686fb6c85 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/793f3bf4a49a4d108609efd686fb6c85
I20260806 04:07:16.546814 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 793f3bf4a49a4d108609efd686fb6c85 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:16.548441 6385 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:16.555899 5862 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.557937 6426 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.559917 5030 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.561568 6417 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:16.563059 4923 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.568738 6417 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.587769 6452 raft_consensus.cc:1050] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:16.590564 6452 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:16.614235 5863 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.616053 6450 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.628033 4923 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.623793 6385 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.696897 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:16.722216 5863 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.730804 5035 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.732501 5030 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.749131 6297 raft_consensus.cc:1050] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:16.750442 4674 catalog_manager.cc:5495] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:16.751750 6297 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:16.758677 4817 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.768913 4817 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:16.780916 6413 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.785480 5035 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:16.787303 6490 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:16.787338 5862 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.791956 6297 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:16.794615 6452 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.801282 4817 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.803195 5866 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:16.819577 6385 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:16.841954 6502 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:16.923655 5032 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:16.919167 6498 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:16.933317 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 67673.8us
I20260806 04:07:16.951421 5867 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:16.955552 6297 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 04:07:17.013360 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 43686.4us
I20260806 04:07:16.939155 5028 raft_consensus.cc:2932] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.089066 5032 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.090742 4674 catalog_manager.cc:5495] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.068723 6501 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.145854 4809 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:17.150859 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 100791us
I20260806 04:07:17.215893 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7cb317a04be74515913f7aee753c193b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:17.233675 6452 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.321165 5862 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:17.355602 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 235480us
W20260806 04:07:17.358038 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 198723us
W20260806 04:07:17.455704 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 233525us
I20260806 04:07:17.507758 5867 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.529635 5029 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.608455 4930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "141037797aa84869bb31e99383874122"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:17.609252 4930 raft_consensus.cc:595] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:17.610379 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cb317a04be74515913f7aee753c193b with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:17.614666 4813 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.635574 6502 raft_consensus.cc:979] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:17.636157 6457 raft_consensus.cc:1067] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:17.640404 4931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54785970285b4054b83ec528e156c25c"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:17.615912 4676 catalog_manager.cc:5495] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:17.641078 4931 raft_consensus.cc:595] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:17.675942 5029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "141037797aa84869bb31e99383874122"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:17.676745 5029 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:17.677254 5029 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.683197 4668 catalog_manager.cc:5495] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.697006 4809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:17.697688 4809 raft_consensus.cc:595] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:17.699215 5029 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:07:17.711392 4924 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:17.714484 5029 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:17.714241 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:17.715974 4930 raft_consensus.cc:3032] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:17.716281 4930 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:17.717061 4930 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:07:17.715960 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:17.720293 4924 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:17.720738 4804 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.720916 4924 raft_consensus.cc:595] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:17.724820 5867 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:17.727531 6457 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:17.727974 6538 tablet_replica.cc:330] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:17.729266 6538 raft_consensus.cc:2229] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:17.729983 6538 raft_consensus.cc:2258] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:17.742246 6457 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.745636 5862 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.747371 6538 ts_tablet_manager.cc:1905] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:17.718410 4930 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.756001 6496 raft_consensus.cc:979] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:17.756593 6496 raft_consensus.cc:1067] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:17.762008 5867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59636
I20260806 04:07:17.767129 5867 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:17.772681 4817 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:17.767668 5867 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.780977 6538 ts_tablet_manager.cc:1918] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:17.781605 6538 log.cc:1196] T 7cb317a04be74515913f7aee753c193b P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/7cb317a04be74515913f7aee753c193b
I20260806 04:07:17.781296 4931 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:17.783849 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 7cb317a04be74515913f7aee753c193b (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:17.791976 5867 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:17.792289 4672 catalog_manager.cc:5495] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.793468 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f75362215d584713a892bd7319a22469 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:17.795603 5867 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:17.795985 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:17.799482 4813 raft_consensus.cc:3032] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:17.799777 4813 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:17.797001 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:17.800643 4927 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.798022 5862 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:17.802341 4813 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:17.807672 4813 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.811494 4927 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:17.802789 4804 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:17.805418 4930 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:17.816375 6413 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:17.817685 6457 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:17.822736 4816 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:17.828850 4809 raft_consensus.cc:2932] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.840302 6413 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:17.851341 4992 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:17.856675 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:17.861017 6536 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:17.861862 6536 raft_consensus.cc:686] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:17.864380 6534 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:17.866631 6534 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:17.870627 4813 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:17.891199 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet f75362215d584713a892bd7319a22469 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:17.895547 6457 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.887166 6534 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } }
I20260806 04:07:17.915545 6503 raft_consensus.cc:979] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:17.914769 5866 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.916077 6503 raft_consensus.cc:1067] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:17.871857 6536 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:17.919328 6452 raft_consensus.cc:979] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:17.919802 6452 raft_consensus.cc:1067] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:17.923452 5860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:45574
I20260806 04:07:17.924074 5860 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:17.924456 5860 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:17.930060 5029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54785970285b4054b83ec528e156c25c"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:17.930790 5029 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:17.937686 5029 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:17.966583 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a3ae3b855e0d4f98a67493c708f71d33 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:17.981216 4813 raft_consensus.cc:2932] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:17.988348 4672 catalog_manager.cc:5495] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:17.989517 4676 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:18.018329 4668 catalog_manager.cc:5495] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:18.032244 5860 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:18.035180 5029 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.038086 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:18.038745 4804 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:18.039531 5028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:18.048933 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:18.049744 4927 raft_consensus.cc:3032] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:18.050105 4927 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:18.051139 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:18.050954 4927 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:18.047753 6544 tablet_replica.cc:330] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:18.053860 4927 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.051995 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:18.054955 4931 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:18.051860 5030 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.055238 4931 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:18.056143 4931 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.057042 4931 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:18.051447 5029 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:18.062892 4927 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:18.053498 5860 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:18.072559 6544 raft_consensus.cc:2229] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:18.073246 5030 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:18.073539 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:18.073463 6544 raft_consensus.cc:2258] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:18.079923 6536 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:18.080456 6536 raft_consensus.cc:686] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:18.081385 6536 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:18.088270 5028 raft_consensus.cc:595] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:18.090044 4931 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:18.091343 4930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:18.092425 4930 raft_consensus.cc:595] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:18.093538 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:18.096099 4804 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:18.096421 6510 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:18.097142 6510 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:18.098275 6510 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.103663 6544 ts_tablet_manager.cc:1905] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:18.105746 4930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:18.106402 4930 raft_consensus.cc:595] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:18.106838 4930 raft_consensus.cc:619] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Rejecting request to transfer leadership because tablet server 99690415a7b3441fbdb5e943667556a4 is not a voter in the active config
I20260806 04:07:18.108814 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260806 04:07:18.122629 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:18.123443 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:18.131603 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ae3b855e0d4f98a67493c708f71d33 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:18.134228 4674 catalog_manager.cc:5495] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 3 to 4, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 4 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:18.143028 4672 catalog_manager.cc:5495] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:18.198553 6544 ts_tablet_manager.cc:1918] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:18.199111 6544 log.cc:1196] T f75362215d584713a892bd7319a22469 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/f75362215d584713a892bd7319a22469
I20260806 04:07:18.200896 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet f75362215d584713a892bd7319a22469 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:18.202139 6544 tablet_replica.cc:330] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:18.204296 6544 raft_consensus.cc:2229] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:18.205896 6544 raft_consensus.cc:2258] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:18.210942 6544 ts_tablet_manager.cc:1905] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:18.238857 6544 ts_tablet_manager.cc:1918] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:18.239364 6544 log.cc:1196] T a3ae3b855e0d4f98a67493c708f71d33 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/a3ae3b855e0d4f98a67493c708f71d33
I20260806 04:07:18.240986 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet a3ae3b855e0d4f98a67493c708f71d33 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:18.254746 6510 raft_consensus.cc:979] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:18.255376 6510 raft_consensus.cc:1067] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:18.257220 4931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:18.257849 4931 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:18.258226 4931 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.280725 4931 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.282871 4931 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:18.284987 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:18.285770 4813 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.287133 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:18.287866 5028 raft_consensus.cc:3032] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:18.288185 5028 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:18.289072 5028 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.290510 5028 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.294064 4813 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:18.295403 4882 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:18.296330 6502 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:18.296897 6502 raft_consensus.cc:686] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:18.297806 6502 consensus_queue.cc:237] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.316957 4672 catalog_manager.cc:5495] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:18.318223 5028 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:18.371258 4931 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:18.372695 6510 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:18.394714 4931 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:18.395007 4804 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:18.396453 6536 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:18.401376 6510 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:18.408535 6413 raft_consensus.cc:979] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:18.416966 6413 raft_consensus.cc:1067] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:18.418983 5028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:18.419606 5028 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:18.420007 5028 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.433470 5028 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.436344 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:18.437050 4813 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.455425 5028 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:18.456423 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:18.463222 4927 raft_consensus.cc:3032] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:18.463707 4927 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:18.464705 4927 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.466825 4927 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:18.461751 4813 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:18.476294 4996 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:18.477425 6534 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:18.477927 6534 raft_consensus.cc:686] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:18.483170 4809 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:18.484875 6536 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:18.486557 4927 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:18.487914 6534 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:18.522492 4931 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:18.527612 6196 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:18.556102 4931 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:18.557895 6537 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:18.570451 4672 catalog_manager.cc:5495] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:18.583165 4809 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:18.589048 6196 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:18.594820 5028 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:18.599071 6537 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:18.682088 4809 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:18.683573 5028 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:18.683928 6503 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:18.685101 6501 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:19.073236 5029 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:19.083768 4995 consensus_peers.cc:487] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57d20919f4994e71841cccb9546444a3. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:19.083803 4931 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:19.087015 6510 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.090531 4804 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:19.090703 4809 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:19.092669 6510 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:19.093501 6505 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:19.115581 4931 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:19.115828 6510 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.120919 4928 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.124271 6583 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.168310 4928 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:19.156376 4809 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.198522 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:19.200287 4674 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 04:07:19.217862 4879 consensus_peers.cc:487] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a878f4d22e44a290e162038ddd83e1. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:19.220631 5028 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:19.221912 4809 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:19.222272 6413 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.223658 6503 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:19.245502 6413 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.249559 4809 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.261113 5030 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
W20260806 04:07:19.267077 4995 consensus_peers.cc:487] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 141037797aa84869bb31e99383874122. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:19.268307 4930 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:07:19.273729 4804 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:07:19.275671 6578 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:19.277771 6578 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:19.305332 4674 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:19.312131 6583 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.334846 4928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:19.335557 4928 raft_consensus.cc:595] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:19.338707 4809 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.345952 5028 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.349998 4928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:19.350751 4928 raft_consensus.cc:595] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:19.362953 5030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:19.363822 5030 raft_consensus.cc:595] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:19.367823 4668 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:19.368549 4927 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:19.364997 4928 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.376231 4809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:19.376866 4809 raft_consensus.cc:595] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:19.388477 4928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:19.389127 4928 raft_consensus.cc:595] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:19.390537 6594 raft_consensus.cc:979] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:19.391176 6594 raft_consensus.cc:1067] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:19.394503 4809 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:07:19.394884 4879 consensus_peers.cc:487] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdc0072b6de7424caf2520a26bdcd0b0. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:19.396344 6503 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.398376 5028 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:19.398869 4928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f75362215d584713a892bd7319a22469"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:19.399184 5030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:19.399943 5030 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:19.400744 6196 raft_consensus.cc:979] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:19.401163 5030 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.401331 6196 raft_consensus.cc:1067] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:19.401096 6503 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.399504 4928 raft_consensus.cc:595] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:19.403671 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 04:07:19.405814 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:19.406719 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:19.414561 4927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:19.419731 4927 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:19.419643 4809 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.416855 6589 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.426232 4927 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.417503 5030 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.431051 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:19.431824 4804 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.448743 4804 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:19.450452 4996 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:19.451606 6196 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:19.452116 6196 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:19.453001 6196 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.450105 4672 catalog_manager.cc:5495] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:19.460044 5030 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:19.461079 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:19.461759 4928 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:19.462069 4928 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:19.462901 4928 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.464089 4928 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.470067 6526 raft_consensus.cc:979] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:19.470960 6526 raft_consensus.cc:1067] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:19.484335 4930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45408
I20260806 04:07:19.484967 4930 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:19.485355 4930 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.493420 4930 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.495334 4930 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:19.498018 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:19.498831 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:19.499472 4809 raft_consensus.cc:3032] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:19.499787 4809 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:19.500916 6413 raft_consensus.cc:979] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:19.500626 4809 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.501405 6589 raft_consensus.cc:1067] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:19.502125 4809 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.505218 4672 catalog_manager.cc:5495] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:19.510699 5862 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:45574
I20260806 04:07:19.511651 5862 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:19.512315 5862 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.502228 5028 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.526417 4927 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.529150 5028 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:19.530467 4876 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:19.530543 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:19.532008 4804 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.533272 4928 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:19.535096 4809 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:19.532011 6589 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:19.540136 6589 raft_consensus.cc:686] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:19.541587 6589 consensus_queue.cc:237] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.547385 4927 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:19.547881 4804 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:19.549605 4882 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:19.550879 6595 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:19.551110 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:19.551697 6595 raft_consensus.cc:686] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:19.551295 5862 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.553385 6595 consensus_queue.cc:237] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.551927 5030 raft_consensus.cc:3032] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:19.561488 5030 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:19.562772 5030 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:19.566529 5030 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.568821 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:19.569621 4804 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.577899 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:19.579748 4928 raft_consensus.cc:3032] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:19.575063 5029 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:19.580538 4928 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:19.597299 5862 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:19.598953 4928 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.599888 5030 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:19.600347 4928 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.607722 4804 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:19.609714 5816 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:19.610728 6537 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:19.611217 6537 raft_consensus.cc:686] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:19.612076 6537 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.614238 4928 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:19.631376 4674 catalog_manager.cc:5495] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:19.642553 6589 raft_consensus.cc:979] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:19.620297 4672 catalog_manager.cc:5495] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:19.643155 6589 raft_consensus.cc:1067] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:19.644949 5860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:45574
I20260806 04:07:19.645551 5860 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:19.645944 5860 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.659662 6610 ts_tablet_manager.cc:927] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:19.689330 6610 tablet_copy_client.cc:316] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:19.689152 5860 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.691241 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 57d20919f4994e71841cccb9546444a3 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:19.691856 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 57d20919f4994e71841cccb9546444a3 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-57d20919f4994e71841cccb9546444a3
I20260806 04:07:19.692238 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:19.693781 4804 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.701596 5049 tablet_copy_source_session.cc:205] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:19.702283 5860 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:19.703194 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:19.707064 4928 raft_consensus.cc:3032] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:19.707424 4928 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:19.705302 4804 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:19.709267 5816 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:19.709017 4928 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.710022 6610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d20919f4994e71841cccb9546444a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:19.710134 6537 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:19.710461 4928 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.710744 6537 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:19.711719 6537 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.701012 4668 catalog_manager.cc:5495] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:19.721181 4928 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:19.727357 6610 tablet_copy_client.cc:791] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:19.727994 6610 tablet_copy_client.cc:655] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:19.732661 4668 catalog_manager.cc:5495] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:19.743340 6613 ts_tablet_manager.cc:927] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:19.745656 6610 tablet_copy_client.cc:531] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:19.752525 6589 raft_consensus.cc:979] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:19.753080 6589 raft_consensus.cc:1067] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:19.755050 5860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f75362215d584713a892bd7319a22469"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:45574
I20260806 04:07:19.755692 5860 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:19.756067 5860 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.756500 6610 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:19.764947 6613 tablet_copy_client.cc:316] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:19.767158 4944 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet 77a878f4d22e44a290e162038ddd83e1 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:19.767786 4944 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet 77a878f4d22e44a290e162038ddd83e1 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-77a878f4d22e44a290e162038ddd83e1
I20260806 04:07:19.776732 4944 tablet_copy_source_session.cc:205] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:19.777598 5860 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.780886 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:19.781513 4804 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.782529 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a878f4d22e44a290e162038ddd83e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:19.788964 4804 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:19.789709 4813 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:19.792026 6610 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:19.793002 6610 tablet_bootstrap.cc:492] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:19.793947 6610 ts_tablet_manager.cc:1397] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.000s
I20260806 04:07:19.793677 6595 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:19.794914 6595 raft_consensus.cc:3032] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:19.795256 6595 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:19.796923 6610 raft_consensus.cc:348] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:19.797837 6610 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:19.798542 6610 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:19.799880 5816 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:19.800741 6537 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:19.801229 6537 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:19.801556 6610 ts_tablet_manager.cc:1428] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.007s user 0.006s sys 0.004s
I20260806 04:07:19.796033 6595 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.802472 6595 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:19.802440 6537 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:19.804903 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-57d20919f4994e71841cccb9546444a3 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:19.805330 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-57d20919f4994e71841cccb9546444a3 on tablet 57d20919f4994e71841cccb9546444a3 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:19.809907 5860 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:19.810729 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:19.822825 4927 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:19.830689 4668 catalog_manager.cc:5495] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:19.846727 6613 tablet_copy_client.cc:791] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:19.847571 6613 tablet_copy_client.cc:655] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:19.851861 6610 ts_tablet_manager.cc:927] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:19.853842 6613 tablet_copy_client.cc:531] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:19.862609 6610 tablet_copy_client.cc:316] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:19.865093 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 141037797aa84869bb31e99383874122 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:19.865912 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 141037797aa84869bb31e99383874122 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-141037797aa84869bb31e99383874122
I20260806 04:07:19.871333 6613 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:19.889081 5049 tablet_copy_source_session.cc:205] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:19.896468 6610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141037797aa84869bb31e99383874122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:19.925208 6610 tablet_copy_client.cc:791] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:19.926074 6610 tablet_copy_client.cc:655] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:19.937057 4804 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:19.939257 6610 tablet_copy_client.cc:531] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:19.940248 6595 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.939386 6613 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:19.943396 6613 tablet_bootstrap.cc:492] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:19.944633 6613 ts_tablet_manager.cc:1397] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.074s user 0.021s sys 0.027s
I20260806 04:07:19.960009 6610 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:19.976522 5032 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:19.976948 6613 raft_consensus.cc:348] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:19.978950 6613 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:19.979946 6613 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:19.983561 6595 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:19.985531 6613 ts_tablet_manager.cc:1428] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.040s user 0.005s sys 0.011s
I20260806 04:07:19.987872 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-77a878f4d22e44a290e162038ddd83e1 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:19.988351 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-77a878f4d22e44a290e162038ddd83e1 on tablet 77a878f4d22e44a290e162038ddd83e1 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:20.011919 4813 raft_consensus.cc:1261] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:20.015331 6597 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:20.017274 6613 ts_tablet_manager.cc:927] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:20.038080 5030 raft_consensus.cc:1261] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:20.040072 6413 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:20.082640 4928 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:20.084492 6196 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:20.239952 4813 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 04:07:20.240587 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 76718.9us
I20260806 04:07:20.210414 6613 tablet_copy_client.cc:316] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
W20260806 04:07:20.398531 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 178441us
I20260806 04:07:20.358281 6536 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260806 04:07:20.518776 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:07:20.541103 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 48315.1us
W20260806 04:07:20.501811 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 291822us
W20260806 04:07:20.584720 6647 log.cc:924] Time spent T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260806 04:07:20.591424 5862 raft_consensus.cc:1203] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:20.601632 5866 raft_consensus.cc:1203] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:20.621572 4944 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet fdc0072b6de7424caf2520a26bdcd0b0 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:20.622318 4944 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet fdc0072b6de7424caf2520a26bdcd0b0 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-fdc0072b6de7424caf2520a26bdcd0b0
I20260806 04:07:20.629366 4930 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:20.632627 4944 tablet_copy_source_session.cc:205] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:20.656859 4930 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:20.663398 6610 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:20.664930 6610 tablet_bootstrap.cc:492] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:20.666110 6610 ts_tablet_manager.cc:1397] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.706s user 0.021s sys 0.039s
I20260806 04:07:20.679134 4926 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:20.679570 6661 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:20.681082 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc0072b6de7424caf2520a26bdcd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:20.683579 6610 raft_consensus.cc:348] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:20.685119 6610 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:20.690111 6661 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:20.704110 4804 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:20.686358 6610 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:20.727648 6610 ts_tablet_manager.cc:1428] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.061s user 0.005s sys 0.010s
I20260806 04:07:20.729954 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-141037797aa84869bb31e99383874122 received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:20.730481 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-141037797aa84869bb31e99383874122 on tablet 141037797aa84869bb31e99383874122 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:20.742197 6613 tablet_copy_client.cc:791] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:20.743178 6613 tablet_copy_client.cc:655] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:20.744508 5866 raft_consensus.cc:1203] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:07:20.753665 6661 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:20.771538 4809 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:20.796977 6613 tablet_copy_client.cc:531] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:20.806147 4804 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:20.825582 6653 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 04:07:20.836485 6650 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260806 04:07:20.846405 4809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:20.847291 4809 raft_consensus.cc:595] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:20.847620 6642 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 04:07:20.883571 5030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:20.884436 5030 raft_consensus.cc:595] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:20.931290 6613 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:20.936471 5030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:20.937312 5030 raft_consensus.cc:595] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:21.053697 5030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:21.055140 5030 raft_consensus.cc:595] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:21.091898 4808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:21.092659 4808 raft_consensus.cc:595] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:21.095947 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:07:21.097803 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:21.098508 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:21.099574 6632 raft_consensus.cc:979] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:21.100379 6632 raft_consensus.cc:1067] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:21.107409 4928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:21.108121 4928 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:21.108753 4928 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:21.125392 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 273614us
I20260806 04:07:21.135744 6413 raft_consensus.cc:1050] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:21.137979 6413 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:21.170985 6684 raft_consensus.cc:1050] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:21.205041 6638 raft_consensus.cc:979] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:21.209501 4928 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:21.214478 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:21.215140 4804 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:21.215047 4928 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:21.216203 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:21.216938 5030 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:21.217428 5030 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:21.218416 5030 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:21.219843 5030 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:21.220647 6651 raft_consensus.cc:1067] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:21.222854 5026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:21.223583 5026 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:21.224009 5026 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:21.232565 5026 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:21.235560 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:21.236173 4928 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:21.245854 5030 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:21.257885 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:21.182904 6684 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:21.264638 4928 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:21.234421 5026 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:21.269196 4804 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:21.270085 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:21.272217 4808 raft_consensus.cc:3032] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:21.272143 4992 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:21.272653 4808 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:21.277973 4808 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:21.282912 6640 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:21.287163 4928 raft_consensus.cc:1226] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:21.287882 6639 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:21.288583 6639 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:21.295931 6196 raft_consensus.cc:979] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:21.301602 6196 raft_consensus.cc:1067] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:21.316521 5867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59650
I20260806 04:07:21.317169 5867 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:21.317091 4804 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:21.317803 5867 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:21.319765 4808 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:21.323163 5026 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:21.332027 5867 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:21.341856 5866 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:21.334985 6595 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260806 04:07:21.344445 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:21.345180 4804 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:21.344666 5867 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:21.346997 6589 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:21.357360 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:21.358152 5026 raft_consensus.cc:3032] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:21.358476 5026 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:21.359546 5026 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:21.360841 4804 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:21.361137 5026 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:21.361934 5816 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:21.363788 6691 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:21.364624 6691 raft_consensus.cc:686] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:07:21.375458 6693 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.085s user 0.002s sys 0.000s
I20260806 04:07:21.383962 6196 raft_consensus.cc:1050] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:21.385141 6665 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260806 04:07:21.388288 5026 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:21.334390 6639 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:21.387141 6196 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:21.394280 6685 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260806 04:07:21.366191 6691 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:21.435120 4672 catalog_manager.cc:5495] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 3 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:21.443048 4808 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:21.437934 6640 raft_consensus.cc:686] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:21.495173 6640 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:21.532565 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 70527.1us
W20260806 04:07:21.593371 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 52514.8us
I20260806 04:07:21.605134 5867 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260806 04:07:21.646783 6708 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260806 04:07:21.654078 4928 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 04:07:21.678576 6699 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
W20260806 04:07:21.747535 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 110445us
I20260806 04:07:21.713555 6684 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
W20260806 04:07:21.803488 6700 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.120s user 0.002s sys 0.002s
I20260806 04:07:21.815193 6694 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260806 04:07:21.880784 4674 catalog_manager.cc:5495] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 04:07:21.901135 6613 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260806 04:07:21.918111 6711 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260806 04:07:21.941977 6613 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:21.943432 6613 tablet_bootstrap.cc:492] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:21.944460 6613 ts_tablet_manager.cc:1397] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 1.013s user 0.023s sys 0.031s
W20260806 04:07:21.961218 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 163147us
I20260806 04:07:21.970247 4809 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 04:07:21.972918 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 167977us
I20260806 04:07:21.993104 4809 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:21.999567 4816 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:22.004482 4809 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:07:21.986295 6638 raft_consensus.cc:979] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:22.078186 6653 raft_consensus.cc:979] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
W20260806 04:07:22.104636 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 124359us
W20260806 04:07:22.112474 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 138807us
I20260806 04:07:22.119369 4808 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 04:07:22.121726 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 125769us
I20260806 04:07:22.122629 5863 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:22.143011 6196 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:22.150336 5026 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:22.182765 6653 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:22.178077 6510 raft_consensus.cc:1067] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:22.190721 6613 raft_consensus.cc:348] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:22.203433 6644 raft_consensus.cc:1067] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:22.203459 6613 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:22.207914 5860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59650
I20260806 04:07:22.208611 5860 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:22.209110 5860 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:22.208834 6613 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:22.215814 6613 ts_tablet_manager.cc:1428] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.271s user 0.010s sys 0.002s
I20260806 04:07:22.217900 6661 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260806 04:07:22.219471 6742 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:22.220254 6742 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:22.282837 6742 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:22.311262 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 173964us
I20260806 04:07:22.335441 6661 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260806 04:07:22.447916 5862 raft_consensus.cc:1203] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:22.451117 4676 catalog_manager.cc:5495] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:22.465122 6503 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:22.424537 5860 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.515661 6661 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:22.516562 6661 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.526624 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:22.531020 5860 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:22.531302 5863 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:22.466495 6503 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.532649 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:22.543206 5032 raft_consensus.cc:3032] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:22.543645 5032 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:22.544575 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:22.544505 5032 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.559286 6661 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:22.561558 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:22.563719 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:22.567276 6661 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:22.542578 6503 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:22.537660 6413 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:22.575233 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 451629us
W20260806 04:07:22.622354 4771 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.002s sys 0.056s
W20260806 04:07:22.623540 4771 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.003s sys 0.056s
W20260806 04:07:22.627532 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 77663.4us
W20260806 04:07:22.627749 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 50157.2us
W20260806 04:07:22.630543 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 112656us
I20260806 04:07:22.637028 5862 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:22.637108 6637 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:22.638500 6637 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.640630 5032 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:22.666348 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.016s sys 0.023s
W20260806 04:07:22.666982 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.017s sys 0.023s
I20260806 04:07:22.668993 6754 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:22.669678 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:22.670080 6754 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.679040 6196 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:22.687121 6637 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:22.686705 6196 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260806 04:07:22.686986 6754 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:22.710433 6196 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:22.711325 6196 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:22.705643 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 392384us
W20260806 04:07:22.729571 4876 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.002s sys 0.012s
W20260806 04:07:22.730094 4876 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.002s sys 0.012s
I20260806 04:07:22.735973 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:22.746749 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 594446us
I20260806 04:07:22.759035 6757 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:22.760146 6757 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.765293 6635 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:22.765398 6684 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:22.765372 6196 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:22.774900 6635 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.777594 6635 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:22.778322 6716 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:22.779079 6716 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.783633 6716 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:22.792112 5032 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:22.789052 6684 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.810092 6684 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:22.828895 6697 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:22.829918 6697 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.833431 6697 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:22.803959 6635 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:22.835155 6635 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.838095 6635 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:22.847391 6665 raft_consensus.cc:1050] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:22.862159 6637 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:22.862977 6637 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:22.854311 5863 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:22.865178 4930 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:07:22.848009 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-fdc0072b6de7424caf2520a26bdcd0b0 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:22.849339 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:22.826413 6196 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:22.867192 4926 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:22.802148 6757 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:22.870237 5812 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:22.871668 4930 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:22.859917 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:22.840699 6765 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260806 04:07:22.866582 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-fdc0072b6de7424caf2520a26bdcd0b0 on tablet fdc0072b6de7424caf2520a26bdcd0b0 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:22.875268 6661 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:22.876191 6721 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:22.878351 6637 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:22.897740 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:22.900195 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:22.901165 6754 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:22.886210 6721 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.864871 6665 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.908067 6640 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:22.908746 6661 raft_consensus.cc:686] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:22.902063 6754 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:22.914063 4926 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:22.919076 4928 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:22.908815 6640 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.932040 6640 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:22.934129 6721 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:22.935531 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:22.936061 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:22.927531 4816 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:22.910374 6661 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:22.939841 6754 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:22.942518 6637 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:22.943205 6637 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:22.945375 6637 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:22.955499 5026 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:22.957401 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:22.957748 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:22.959043 4672 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46900 (request call id 239) took 1533 ms. Trace:
I20260806 04:07:22.959249 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:22.959754 4930 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:22.970796 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:22.971364 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:22.971767 5028 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:22.972321 4805 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:22.975224 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:22.983996 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:22.978423 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:22.995139 6637 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:22.995289 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:22.995893 6637 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.018142 4674 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56998 (request call id 115) took 1399 ms. Trace:
I20260806 04:07:23.035084 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.045702 6635 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:23.046612 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.046557 6635 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.048144 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.050380 6640 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:23.050788 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:23.051074 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:23.051816 6635 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:23.047237 4676 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46902 (request call id 247) took 1074 ms. Trace:
I20260806 04:07:23.052620 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.056046 4882 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:23.000299 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 56671.9us
I20260806 04:07:23.018669 4674 rpcz_store.cc:276] 0806 04:07:21.612962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:21.613323 (+ 361us) service_pool.cc:227] Handling call
0806 04:07:23.012576 (+1399253us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:22.598297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:22.740067 (+141770us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:22.740171 (+ 104us) write_op.cc:435] Acquired schema lock
0806 04:07:22.740227 (+ 56us) tablet.cc:660] Decoding operations
0806 04:07:22.740916 (+ 689us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:22.741038 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:22.741117 (+ 79us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:07:22.741631 (+ 514us) tablet.cc:699] Row locks acquired
0806 04:07:22.741680 (+ 49us) write_op.cc:260] PREPARE: finished
0806 04:07:22.741971 (+ 291us) write_op.cc:270] Start()
0806 04:07:22.742183 (+ 212us) write_op.cc:276] Timestamp: P: 1785989242741881 usec, L: 0
0806 04:07:22.742244 (+ 61us) op_driver.cc:331] REPLICATION: starting
0806 04:07:22.756975 (+ 14731us) log.cc:842] Serialized 533 byte log entry
0806 04:07:22.883279 (+126304us) op_driver.cc:444] REPLICATION: finished
0806 04:07:22.949416 (+ 66137us) write_op.cc:301] APPLY: starting
0806 04:07:22.949492 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:22.949586 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:22.949710 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:22.949925 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:22.950112 (+ 187us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:07:22.950164 (+ 52us) write_op.cc:312] APPLY: finished
0806 04:07:22.950361 (+ 197us) log.cc:842] Serialized 23 byte log entry
0806 04:07:22.968451 (+ 18090us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:22.976681 (+ 8230us) write_op.cc:454] Released schema lock
0806 04:07:22.979041 (+ 2360us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:22.995283 (+ 16242us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:22.995331 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":61744,"apply.run_cpu_time_us":2260,"apply.run_wall_time_us":46220,"num_ops":1,"prepare.queue_time_us":698286,"prepare.run_cpu_time_us":6217,"prepare.run_wall_time_us":159204,"replication_time_us":130599,"thread_start_us":24137,"threads_started":1}]]}
I20260806 04:07:23.051103 6640 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.066910 6684 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:23.052523 4676 rpcz_store.cc:276] 0806 04:07:21.973122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:21.973253 (+ 131us) service_pool.cc:227] Handling call
0806 04:07:23.047061 (+1073808us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:22.757929 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:22.759003 (+ 1074us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:22.759046 (+ 43us) write_op.cc:435] Acquired schema lock
0806 04:07:22.759066 (+ 20us) tablet.cc:660] Decoding operations
0806 04:07:22.759647 (+ 581us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:22.759704 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:22.759735 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:07:22.761731 (+ 1996us) tablet.cc:699] Row locks acquired
0806 04:07:22.761765 (+ 34us) write_op.cc:260] PREPARE: finished
0806 04:07:22.762002 (+ 237us) write_op.cc:270] Start()
0806 04:07:22.762142 (+ 140us) write_op.cc:276] Timestamp: P: 1785989242761939 usec, L: 0
0806 04:07:22.762161 (+ 19us) op_driver.cc:331] REPLICATION: starting
0806 04:07:22.763058 (+ 897us) log.cc:842] Serialized 538 byte log entry
0806 04:07:22.985318 (+222260us) op_driver.cc:444] REPLICATION: finished
0806 04:07:22.995929 (+ 10611us) write_op.cc:301] APPLY: starting
0806 04:07:22.996034 (+ 105us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:22.996176 (+ 142us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:22.996364 (+ 188us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:22.996726 (+ 362us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:22.996980 (+ 254us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:07:22.997069 (+ 89us) write_op.cc:312] APPLY: finished
0806 04:07:22.997355 (+ 286us) log.cc:842] Serialized 23 byte log entry
0806 04:07:22.997635 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:22.997774 (+ 139us) write_op.cc:454] Released schema lock
0806 04:07:22.997845 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:23.011126 (+ 13281us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:23.014381 (+ 3255us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":5903,"apply.run_cpu_time_us":2783,"apply.run_wall_time_us":38969,"num_ops":1,"prepare.queue_time_us":250863,"prepare.run_cpu_time_us":3788,"prepare.run_wall_time_us":10217,"replication_time_us":213598}]]}
I20260806 04:07:23.075296 6753 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:23.076961 5866 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:22.959429 4672 rpcz_store.cc:276] 0806 04:07:21.425054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:21.425206 (+ 152us) service_pool.cc:227] Handling call
0806 04:07:22.958827 (+1533621us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:22.094161 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:22.150461 (+ 56300us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:22.150495 (+ 34us) write_op.cc:435] Acquired schema lock
0806 04:07:22.150506 (+ 11us) tablet.cc:660] Decoding operations
0806 04:07:22.150890 (+ 384us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:22.150945 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:22.150969 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:07:22.151149 (+ 180us) tablet.cc:699] Row locks acquired
0806 04:07:22.151163 (+ 14us) write_op.cc:260] PREPARE: finished
0806 04:07:22.151302 (+ 139us) write_op.cc:270] Start()
0806 04:07:22.151408 (+ 106us) write_op.cc:276] Timestamp: P: 1785989242151262 usec, L: 0
0806 04:07:22.151424 (+ 16us) op_driver.cc:331] REPLICATION: starting
0806 04:07:22.152137 (+ 713us) log.cc:842] Serialized 538 byte log entry
0806 04:07:22.775560 (+623423us) op_driver.cc:444] REPLICATION: finished
0806 04:07:22.940682 (+165122us) write_op.cc:301] APPLY: starting
0806 04:07:22.940815 (+ 133us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:22.940942 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:22.945248 (+ 4306us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:22.945580 (+ 332us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:22.945944 (+ 364us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:07:22.946008 (+ 64us) write_op.cc:312] APPLY: finished
0806 04:07:22.946329 (+ 321us) log.cc:842] Serialized 23 byte log entry
0806 04:07:22.948301 (+ 1972us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:22.948421 (+ 120us) write_op.cc:454] Released schema lock
0806 04:07:22.948483 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:22.948861 (+ 378us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:22.948898 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":141913,"apply.run_cpu_time_us":2572,"apply.run_wall_time_us":8527,"num_ops":1,"prepare.queue_time_us":544380,"prepare.run_cpu_time_us":30916,"prepare.run_wall_time_us":527305,"raft.queue_time_us":398167,"raft.run_cpu_time_us":41199,"raft.run_wall_time_us":189940,"replication_time_us":624002,"spinlock_wait_cycles":4864,"thread_start_us":892337,"threads_started":4,"wal-append.queue_time_us":514348}]]}
I20260806 04:07:23.019450 5032 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:23.095930 5032 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:23.096737 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.101636 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.105741 5032 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.080580 6753 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260806 04:07:23.116685 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.117288 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.121078 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.122181 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.106218 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:23.135236 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:23.119338 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.142118 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:23.143196 4924 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:23.149559 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.155230 4668 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46886 (request call id 249) took 1538 ms. Trace:
I20260806 04:07:23.138435 6640 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:23.142393 6637 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:23.158793 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.160969 4814 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.155558 4668 rpcz_store.cc:276] 0806 04:07:21.501852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:21.501986 (+ 134us) service_pool.cc:227] Handling call
0806 04:07:23.026364 (+1524378us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
I20260806 04:07:23.168541 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.165052 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:23.172194 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.179522 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.181790 4882 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:23.184257 6757 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.185599 4996 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.152637 6665 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260806 04:07:23.186172 6503 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:23.197081 6503 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:23.188203 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.197489 6503 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:23.188553 6731 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.194950 6635 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.178249 6684 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:23.204931 6684 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.180493 6640 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.210681 4816 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:23.219017 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.219998 4768 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:23.221365 6637 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:23.221828 6749 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.222724 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:23.223568 6756 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:23.223974 6756 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:23.224398 6756 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:23.226236 6503 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.229117 6503 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:23.237382 6684 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:23.239825 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:23.245221 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 93699.6us
I20260806 04:07:23.251087 4876 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.254048 4876 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.255662 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.256222 6765 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.257541 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.259017 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.259222 6757 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:23.261620 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:23.260509 6791 log.cc:924] Time spent T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 04:07:23.266498 6684 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.281749 5032 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.289924 5038 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:23.290402 6697 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.276196 6640 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260806 04:07:23.319576 6756 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.326866 4668 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.333226 4992 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.341931 5032 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:23.342960 6640 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:23.350215 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:23.351289 5036 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:23.351485 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:23.381042 6640 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.356736 5029 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.384123 6640 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:23.400022 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 67054.9us
I20260806 04:07:23.400760 4876 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.400882 4992 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.393483 4668 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:23.403185 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:23.406378 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:23.408013 4876 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.404729 4668 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.402998 6635 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.417716 4992 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.418033 4804 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:23.414093 6753 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.409368 6757 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260806 04:07:23.419842 4992 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.420284 6731 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.421047 4992 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.421047 6731 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:23.413576 6716 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:23.422917 4814 raft_consensus.cc:3032] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:23.423290 4814 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:23.423398 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:23.424156 4928 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:23.423640 5863 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.424306 4814 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.431282 6196 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.433753 6657 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:23.456705 4814 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.468356 4928 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:23.471549 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:23.473308 4768 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:23.474476 4814 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:23.477705 6196 consensus_queue.cc:1046] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:23.478605 6196 raft_consensus.cc:3032] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:23.478989 6196 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:23.479821 6196 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.481117 6196 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:23.481594 4924 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260806 04:07:23.482059 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:23.478768 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:23.483098 4927 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:23.487576 4926 raft_consensus.cc:1226] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:23.488240 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:23.492365 4996 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:23.494181 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:23.491830 4931 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.474327 6756 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:23.491842 6754 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.504638 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:23.506968 6691 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:23.501583 6716 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:23.512048 6666 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.515097 6716 raft_consensus.cc:686] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:23.516172 6716 consensus_queue.cc:237] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.516268 6753 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.527792 4992 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.530078 4992 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:23.532335 5036 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:23.531327 5866 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.539350 4931 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:23.532006 6640 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.529665 6684 consensus_queue.cc:1046] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260806 04:07:23.547341 6759 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:23.541626 6661 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260806 04:07:23.550987 6684 raft_consensus.cc:3032] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:23.551374 6684 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:23.552217 6684 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.553575 6684 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:23.563972 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:23.565004 4923 raft_consensus.cc:595] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:23.619717 5030 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:23.621592 4814 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.637535 4805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:23.650825 4805 raft_consensus.cc:595] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:23.657625 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 163512us
I20260806 04:07:23.659868 4931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8570417a88db412294c21c9108061c30"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:23.658829 5038 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.660440 4931 raft_consensus.cc:595] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:23.671599 6749 raft_consensus.cc:979] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:23.672298 6818 raft_consensus.cc:1067] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:23.694564 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:23.694976 5028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:23.695581 5028 raft_consensus.cc:595] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:23.697117 6749 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:23.697665 6749 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:23.698654 6749 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.705423 4814 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:23.706103 4814 raft_consensus.cc:595] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:23.709551 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:07:23.687578 5030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:23.710316 5030 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:23.710757 5030 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.711404 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:23.716753 5030 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.726266 5030 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:23.727519 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:23.725221 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:23.728240 4814 raft_consensus.cc:3032] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:23.728677 4931 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.728684 4814 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:23.729776 4814 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.724892 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:23.735978 4931 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:23.737104 4992 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:23.738090 6632 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:23.738622 6632 raft_consensus.cc:686] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:23.739481 6632 consensus_queue.cc:237] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.742578 4669 catalog_manager.cc:5495] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.745035 4814 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.760340 4814 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:23.759905 5030 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.763302 4669 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.769029 5863 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:23.770463 4805 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:23.783670 6767 raft_consensus.cc:979] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:23.784149 6767 raft_consensus.cc:1067] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:23.784497 6749 raft_consensus.cc:979] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:23.785005 6749 raft_consensus.cc:1067] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:23.785394 6759 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:23.787623 5866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59636
I20260806 04:07:23.788260 5866 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:23.788604 5866 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.789803 5038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:23.790040 6196 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:23.793021 4925 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.790390 5038 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:23.796326 5038 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.799196 6196 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.812857 4805 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.813957 4930 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.816464 4668 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 4 to 3, term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from VOTER to NON_VOTER, now has a pending config: VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67), VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66), VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65), VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.818598 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 57d20919f4994e71841cccb9546444a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:23.821245 5863 raft_consensus.cc:2932] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.828759 5867 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:23.832512 6757 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:23.835433 5867 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:23.838433 5038 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.849754 5038 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:23.840701 6757 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:23.851353 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:23.851982 4804 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.845322 6665 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.851432 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:23.844055 5866 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.860783 4930 raft_consensus.cc:3032] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:23.861195 4930 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:23.861810 5866 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:23.849360 5867 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.840023 4814 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:23.863893 5030 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:07:23.864714 6503 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.866397 6753 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:23.866397 5862 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.867611 4930 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.868999 4930 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.874605 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:23.875447 4804 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:23.875618 4925 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.875998 6757 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:23.879943 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fdc0072b6de7424caf2520a26bdcd0b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:23.880885 4651 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 77a878f4d22e44a290e162038ddd83e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:23.881420 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:23.883852 4805 raft_consensus.cc:3032] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:23.885782 4805 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:23.889338 5030 raft_consensus.cc:2932] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.892367 4996 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:23.895397 6759 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:23.895870 6759 raft_consensus.cc:686] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:23.896770 6759 consensus_queue.cc:237] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.907044 6635 raft_consensus.cc:979] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:23.907593 6635 raft_consensus.cc:1067] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:23.907387 4805 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.908886 4805 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.909567 4920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:23.910147 4920 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:23.910534 4920 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.915325 4814 raft_consensus.cc:2932] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.920928 4920 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.926225 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:23.926888 4810 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.929687 4805 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:23.931200 5816 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:23.933172 6661 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:23.933688 6661 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:23.934715 4810 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:23.935943 4882 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:23.936610 4669 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67), VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66), VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65), VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.938105 6503 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:23.939762 6503 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:23.940771 6503 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.934573 6661 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:23.884137 4925 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:23.951858 4920 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:23.952826 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:23.955945 5029 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:23.956300 5029 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:23.957646 4669 catalog_manager.cc:5495] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:23.957370 5029 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:23.959038 5029 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:23.965391 4669 catalog_manager.cc:5495] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:23.969657 5029 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:23.977490 4930 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:23.980598 4674 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 3 to 4, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 1 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.985826 4672 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 3 to 4, VOTER df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) evicted. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:23.991223 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d20919f4994e71841cccb9546444a3 with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:23.993063 6834 tablet_replica.cc:330] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:23.994210 6834 raft_consensus.cc:2229] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:23.995059 6834 raft_consensus.cc:2258] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:24.000385 6834 ts_tablet_manager.cc:1905] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:24.008538 4674 catalog_manager.cc:5495] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: config changed from index 3 to 4, VOTER 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) evicted. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.010598 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a878f4d22e44a290e162038ddd83e1 with delete_type TABLET_DATA_TOMBSTONED (TS df2f1e7811114b8a995f6771fb6a6c8a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33546
I20260806 04:07:24.027097 4672 catalog_manager.cc:5495] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:24.031208 6834 ts_tablet_manager.cc:1918] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:24.032262 5019 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc0072b6de7424caf2520a26bdcd0b0 with delete_type TABLET_DATA_TOMBSTONED (TS 99690415a7b3441fbdb5e943667556a4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34040
I20260806 04:07:24.040159 6835 tablet_replica.cc:330] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: stopping tablet replica
I20260806 04:07:24.041172 6834 log.cc:1196] T 57d20919f4994e71841cccb9546444a3 P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/57d20919f4994e71841cccb9546444a3
I20260806 04:07:24.042955 6835 raft_consensus.cc:2229] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:24.043851 6835 raft_consensus.cc:2258] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:24.041510 6836 tablet_replica.cc:330] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: stopping tablet replica
I20260806 04:07:24.044075 4668 catalog_manager.cc:5495] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:24.046634 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 57d20919f4994e71841cccb9546444a3 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:24.054337 4674 catalog_manager.cc:5495] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:24.056901 6836 raft_consensus.cc:2229] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:24.057857 6836 raft_consensus.cc:2258] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:24.061264 6835 ts_tablet_manager.cc:1905] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:24.083273 4930 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.063907 6836 ts_tablet_manager.cc:1905] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:24.072582 6757 raft_consensus.cc:979] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:24.084065 6665 raft_consensus.cc:1067] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:24.085060 6759 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.085536 5029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8570417a88db412294c21c9108061c30"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:24.086202 5029 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:24.086553 5029 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:24.092576 5029 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:24.095759 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:24.096417 4813 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:24.106308 4813 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:24.107784 4996 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:24.108739 6196 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:24.109236 6196 raft_consensus.cc:686] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:24.113472 4813 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.114800 5030 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:24.110083 6196 consensus_queue.cc:237] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:24.116712 6526 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:24.117063 6632 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.117887 6836 ts_tablet_manager.cc:1918] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:24.118366 6836 log.cc:1196] T fdc0072b6de7424caf2520a26bdcd0b0 P 99690415a7b3441fbdb5e943667556a4: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-2-root/wals/fdc0072b6de7424caf2520a26bdcd0b0
I20260806 04:07:24.120543 6835 ts_tablet_manager.cc:1918] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:07:24.121019 6835 log.cc:1196] T 77a878f4d22e44a290e162038ddd83e1 P df2f1e7811114b8a995f6771fb6a6c8a: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-0-root/wals/77a878f4d22e44a290e162038ddd83e1
I20260806 04:07:24.126261 5029 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:24.128090 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:24.130759 4920 raft_consensus.cc:3032] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:24.131138 4920 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:24.131919 4920 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:24.133160 4920 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:24.138980 4651 catalog_manager.cc:4841] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): tablet fdc0072b6de7424caf2520a26bdcd0b0 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:24.140097 4657 catalog_manager.cc:4841] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): tablet 77a878f4d22e44a290e162038ddd83e1 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:24.138533 4813 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.154733 6196 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.154860 4674 catalog_manager.cc:5495] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:24.169433 4920 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:24.184485 4926 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:24.187110 6526 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:07:24.196760 4926 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.205660 5863 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:24.205827 6694 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:24.229624 5863 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:24.231138 6526 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:24.244585 6818 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.264616 4920 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.347334 4804 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:24.351954 5029 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:24.353663 6851 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:24.362591 5867 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:24.364317 6818 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:24.371083 6824 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.375943 5029 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.377442 5867 raft_consensus.cc:2932] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.382438 4657 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 141037797aa84869bb31e99383874122 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:07:24.387596 4920 raft_consensus.cc:1261] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.390214 6694 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:24.401149 4674 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 4 to 6, VOTER 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) evicted. New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:24.411389 4805 raft_consensus.cc:1261] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.412864 6196 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.414463 4913 tablet_service.cc:1514] Processing DeleteTablet for tablet 141037797aa84869bb31e99383874122 with delete_type TABLET_DATA_TOMBSTONED (TS 22fe557f852d4415a96fd683f970cc24 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45392
I20260806 04:07:24.415481 6834 tablet_replica.cc:330] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: stopping tablet replica
I20260806 04:07:24.416363 6834 raft_consensus.cc:2229] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:07:24.417016 6834 raft_consensus.cc:2258] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:07:24.426368 6834 ts_tablet_manager.cc:1905] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:07:24.446328 6834 ts_tablet_manager.cc:1918] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260806 04:07:24.446723 6834 log.cc:1196] T 141037797aa84869bb31e99383874122 P 22fe557f852d4415a96fd683f970cc24: Deleting WAL directory at /tmp/dist-test-taskyHHbI8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785989141369922-3041-0/minicluster-data/ts-1-root/wals/141037797aa84869bb31e99383874122
I20260806 04:07:24.448037 4651 catalog_manager.cc:4841] TS 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): tablet 141037797aa84869bb31e99383874122 (table test-workload [id=0735bc8993c646e89c81220829e37a88]) successfully deleted
I20260806 04:07:24.490628 4926 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:24.492070 6691 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:24.501178 4813 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:24.502714 6691 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:24.530069 5029 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.531987 6503 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.552584 4813 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.554085 6757 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.555962 4805 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.564437 6196 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.570899 4926 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:24.572376 6196 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:24.893299 6756 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:24.893865 6756 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:24.895859 6756 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:24.900095 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:24.901329 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:24.902135 4926 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:24.901438 5038 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:24.905531 4768 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:24.906704 6851 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:24.907083 6851 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:24.907459 6851 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:24.914595 6851 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:24.917833 6851 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:24.918462 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:24.919317 4926 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:24.919298 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:24.925678 5038 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:24.926578 4768 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:24.927327 6851 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:24.927817 6851 raft_consensus.cc:686] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:24.928469 6851 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:24.938854 4672 catalog_manager.cc:5495] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:25.097498 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:25.139386 4813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "141037797aa84869bb31e99383874122"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:25.140033 4813 raft_consensus.cc:595] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:25.142611 4920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:25.143345 4920 raft_consensus.cc:595] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:25.145799 5038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:25.146188 5038 raft_consensus.cc:595] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:25.148288 4920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:25.148795 4920 raft_consensus.cc:595] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:25.149809 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:07:25.151134 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:25.151644 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:25.451274 4923 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:25.461165 6879 raft_consensus.cc:979] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:25.464743 6882 raft_consensus.cc:1067] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:25.471246 6884 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:25.493808 6757 raft_consensus.cc:979] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:25.494324 6757 raft_consensus.cc:1067] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:25.504226 4804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:25.504837 4804 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:25.505203 4804 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:25.507472 4814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:25.508133 4814 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:25.508492 4814 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:25.520776 4804 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:25.523834 4804 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:25.525604 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:25.526342 4931 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:25.526612 4931 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:25.536700 4814 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:25.535892 4931 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:25.542346 4931 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:25.544747 4814 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:25.543263 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:25.547670 4923 raft_consensus.cc:3032] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:25.548038 4923 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:25.548815 4923 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:25.550067 4923 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:25.556748 4931 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:25.557976 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:25.559259 6877 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:25.559789 6877 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:25.560663 6877 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:25.601605 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:25.602299 5862 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:25.614564 5862 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:25.619724 5030 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:25.632498 4926 consensus_queue.cc:237] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:25.656208 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:25.657048 5038 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:25.622347 4923 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:25.659541 4768 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:25.661098 6851 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:25.661911 6851 raft_consensus.cc:686] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:25.662884 6851 consensus_queue.cc:237] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:25.677341 4672 catalog_manager.cc:5495] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:25.689746 6877 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:07:25.690357 4879 consensus_peers.cc:487] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e4a7d4c7807476384d5ce509d4029f9. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:25.693151 4814 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:25.694891 6665 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:25.698068 6756 raft_consensus.cc:979] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:25.698524 6756 raft_consensus.cc:1067] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:25.698738 5030 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:25.700130 6665 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:25.703433 5038 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:25.721432 6665 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.725277 6891 raft_consensus.cc:979] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:25.725800 6891 raft_consensus.cc:1067] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:25.724905 4814 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.733253 4672 catalog_manager.cc:5495] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:25.735029 4926 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:25.735654 4926 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:25.736027 4926 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:25.743424 5030 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.742175 4926 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:25.747370 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:25.747941 4804 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:25.748870 5029 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:25.752094 4669 catalog_manager.cc:5495] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:25.758509 5028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "141037797aa84869bb31e99383874122"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:25.759162 5028 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:25.759526 5028 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:25.766178 5028 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:25.769043 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:25.769709 5862 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:25.772894 4804 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:25.774537 4882 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:25.775408 6665 raft_consensus.cc:2781] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:25.775882 6665 raft_consensus.cc:686] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:25.781330 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:25.783936 4814 raft_consensus.cc:3032] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:25.784237 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:25.784370 4814 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:25.784890 5037 raft_consensus.cc:3032] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:25.783326 5862 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:25.785107 5028 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:25.785229 5037 raft_consensus.cc:729] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:25.785651 4814 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:25.786113 5037 consensus_queue.cc:260] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:25.787093 4814 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:25.787595 5037 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:25.759469 4926 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:25.794627 4995 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:25.795491 6891 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:25.795670 4814 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:25.796191 6891 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:25.797248 5037 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:25.797197 6891 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:25.782621 6665 consensus_queue.cc:237] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:25.825065 4931 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260806 04:07:25.825419 4995 consensus_peers.cc:487] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 997e4d9ff3ca465199a888e8af23ed7b. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:25.825536 4814 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:07:25.826551 6632 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:25.827570 6892 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:25.863858 4669 catalog_manager.cc:5495] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:25.868150 6892 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.876379 4931 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.879230 4814 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.892356 4926 consensus_queue.cc:237] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:25.892880 4672 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 3 to 4, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 4 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:25.902344 4669 catalog_manager.cc:5495] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 3, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
W20260806 04:07:25.925588 4879 consensus_peers.cc:487] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eaa370887c344934b64e0b3f59c6a3b0. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:25.926107 5030 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:25.926477 4814 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:07:25.927444 6503 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:25.928371 6879 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:25.937999 6503 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.940284 5030 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.951957 4804 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:25.944967 4814 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.957080 4669 catalog_manager.cc:5495] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 2, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 04:07:25.961139 4770 consensus_peers.cc:487] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a -> Peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Couldn't send request to peer d7c76b6dcd8945cd818b8107caf4a174. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 086f8af9433f44aa8aa4fccbfa90eb02. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:25.963565 5038 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:07:25.967633 4926 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:07:25.969782 6877 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:25.980247 6877 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:25.982625 6851 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:25.990296 4926 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:26.022447 5030 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:26.029346 4672 catalog_manager.cc:5495] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index -1 to 4, NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) added. New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } } }
I20260806 04:07:26.090093 5030 raft_consensus.cc:1261] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:26.091908 6851 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:26.113631 4923 raft_consensus.cc:1261] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:26.119410 6889 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:26.141273 4926 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:26.143452 5862 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:26.145072 6888 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:26.156118 6888 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:26.231491 6932 ts_tablet_manager.cc:927] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:26.243778 6932 tablet_copy_client.cc:316] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:26.247776 4944 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet 2e4a7d4c7807476384d5ce509d4029f9 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:26.248704 4944 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet 2e4a7d4c7807476384d5ce509d4029f9 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-2e4a7d4c7807476384d5ce509d4029f9
I20260806 04:07:26.260948 4944 tablet_copy_source_session.cc:205] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:26.267086 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a7d4c7807476384d5ce509d4029f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:26.272076 5867 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:07:26.273923 6891 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:07:26.282220 6938 ts_tablet_manager.cc:927] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:26.304689 6938 tablet_copy_client.cc:316] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.67:43701
I20260806 04:07:26.309111 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:26.310969 5049 tablet_copy_service.cc:140] P 99690415a7b3441fbdb5e943667556a4: Received BeginTabletCopySession request for tablet 997e4d9ff3ca465199a888e8af23ed7b from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:34146)
I20260806 04:07:26.311872 5049 tablet_copy_service.cc:161] P 99690415a7b3441fbdb5e943667556a4: Beginning new tablet copy session on tablet 997e4d9ff3ca465199a888e8af23ed7b from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:34146: session id = d7c76b6dcd8945cd818b8107caf4a174-997e4d9ff3ca465199a888e8af23ed7b
I20260806 04:07:26.335986 6932 tablet_copy_client.cc:791] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:26.336964 6932 tablet_copy_client.cc:655] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:26.340749 4804 raft_consensus.cc:1261] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:26.344014 6879 consensus_queue.cc:1035] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:26.347126 6932 tablet_copy_client.cc:531] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:26.367103 4804 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:07:26.385284 6632 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:07:26.395761 5038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54785970285b4054b83ec528e156c25c"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:26.396632 5038 raft_consensus.cc:595] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:26.416899 5038 raft_consensus.cc:1261] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:26.417917 4804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:26.418447 4804 raft_consensus.cc:595] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:26.421847 6879 consensus_queue.cc:1035] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20260806 04:07:26.433915 4962 debug-util.cc:398] Leaking SignalData structure 0x7b0800170b60 after lost signal to thread 3045
I20260806 04:07:26.434899 6942 ts_tablet_manager.cc:927] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
W20260806 04:07:26.435622 4962 debug-util.cc:398] Leaking SignalData structure 0x7b080016d240 after lost signal to thread 4724
W20260806 04:07:26.444067 4962 debug-util.cc:398] Leaking SignalData structure 0x7b080031b280 after lost signal to thread 4853
W20260806 04:07:26.445608 4962 debug-util.cc:398] Leaking SignalData structure 0x7b0800343ea0 after lost signal to thread 4966
W20260806 04:07:26.451310 4962 debug-util.cc:398] Leaking SignalData structure 0x7b0800366a80 after lost signal to thread 5070
W20260806 04:07:26.457818 4962 debug-util.cc:398] Leaking SignalData structure 0x7b0800366000 after lost signal to thread 5901
I20260806 04:07:26.460685 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:26.461725 4923 raft_consensus.cc:595] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:26.467273 6943 ts_tablet_manager.cc:927] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Initiating tablet copy from peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:26.471349 5049 tablet_copy_source_session.cc:205] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:26.479346 6932 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:26.480957 6938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997e4d9ff3ca465199a888e8af23ed7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:26.540102 6942 tablet_copy_client.cc:316] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.65:43865
I20260806 04:07:26.537042 4805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:26.573084 4831 tablet_copy_service.cc:140] P df2f1e7811114b8a995f6771fb6a6c8a: Received BeginTabletCopySession request for tablet 086f8af9433f44aa8aa4fccbfa90eb02 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:53354)
I20260806 04:07:26.556005 4805 raft_consensus.cc:595] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:26.578150 4831 tablet_copy_service.cc:161] P df2f1e7811114b8a995f6771fb6a6c8a: Beginning new tablet copy session on tablet 086f8af9433f44aa8aa4fccbfa90eb02 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:53354: session id = d7c76b6dcd8945cd818b8107caf4a174-086f8af9433f44aa8aa4fccbfa90eb02
I20260806 04:07:26.583005 6877 raft_consensus.cc:979] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:26.578168 6943 tablet_copy_client.cc:316] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.66:34857
I20260806 04:07:26.618461 6877 raft_consensus.cc:1067] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:26.605398 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:07:26.623127 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:26.624032 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:26.650530 6938 tablet_copy_client.cc:791] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:26.651803 6938 tablet_copy_client.cc:655] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:26.656188 4944 tablet_copy_service.cc:140] P 22fe557f852d4415a96fd683f970cc24: Received BeginTabletCopySession request for tablet eaa370887c344934b64e0b3f59c6a3b0 from peer d7c76b6dcd8945cd818b8107caf4a174 ({username='slave'} at 127.0.0.1:55924)
I20260806 04:07:26.657265 4944 tablet_copy_service.cc:161] P 22fe557f852d4415a96fd683f970cc24: Beginning new tablet copy session on tablet eaa370887c344934b64e0b3f59c6a3b0 from peer d7c76b6dcd8945cd818b8107caf4a174 at {username='slave'} at 127.0.0.1:55924: session id = d7c76b6dcd8945cd818b8107caf4a174-eaa370887c344934b64e0b3f59c6a3b0
W20260806 04:07:26.686842 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 110840us
W20260806 04:07:26.701490 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 69587.9us
I20260806 04:07:26.701587 4831 tablet_copy_source_session.cc:205] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:26.717657 6942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086f8af9433f44aa8aa4fccbfa90eb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:26.752995 4944 tablet_copy_source_session.cc:205] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:07:26.756851 6950 raft_consensus.cc:979] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:26.780179 6950 raft_consensus.cc:1067] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:26.760594 5029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:26.781744 5029 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:26.782101 5029 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:26.773715 6938 tablet_copy_client.cc:531] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:26.775676 6893 raft_consensus.cc:979] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:26.787745 6893 raft_consensus.cc:1067] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:26.795101 4931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54785970285b4054b83ec528e156c25c"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:26.795846 4931 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:26.796264 4931 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:26.817847 4931 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:26.822150 4931 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:26.822348 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:26.823122 5036 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:26.823411 5036 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:26.824155 5036 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:26.825206 5036 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:26.827919 6942 tablet_copy_client.cc:791] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:26.831779 6942 tablet_copy_client.cc:655] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:26.833742 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:26.834339 4814 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:26.843278 5036 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:26.846285 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:26.847729 6665 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:26.848270 6665 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:26.849216 6665 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:26.851042 4931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45408
I20260806 04:07:26.852882 4931 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:26.853271 4931 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:26.860540 4814 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:26.861281 4816 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:26.870252 4931 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:26.872051 6942 tablet_copy_client.cc:531] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:26.889827 4931 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:26.889302 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:26.890483 4816 raft_consensus.cc:3032] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:26.891067 4816 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:26.891866 4816 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:26.893378 4816 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:26.807868 5029 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:26.898504 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:26.907058 5860 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:26.909965 6938 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:26.898896 6942 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:26.911441 4674 catalog_manager.cc:5495] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 4 to 5, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 5 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:26.915149 5029 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:26.916476 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:26.916672 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:26.917354 4814 raft_consensus.cc:3032] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:26.918686 4814 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:26.920342 4931 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:26.921345 4816 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:26.920681 4814 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:26.924440 4814 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:26.927090 5860 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:26.928426 4879 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:26.930644 6665 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:26.932078 6665 raft_consensus.cc:686] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:26.933465 6943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa370887c344934b64e0b3f59c6a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:07:26.934055 6665 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:26.942760 4931 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:26.944046 4992 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:26.944959 6956 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:26.946350 6956 raft_consensus.cc:686] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:26.947291 6956 consensus_queue.cc:237] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:26.960806 6932 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:26.965576 4814 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:26.963228 4669 catalog_manager.cc:5495] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:26.974466 6932 tablet_bootstrap.cc:492] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:26.975385 6932 ts_tablet_manager.cc:1397] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.497s user 0.021s sys 0.016s
I20260806 04:07:27.006193 6932 raft_consensus.cc:348] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.007175 6932 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:27.007836 6932 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.043320 6503 raft_consensus.cc:979] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:27.043960 6879 raft_consensus.cc:1067] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:27.053222 4814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:27.053845 4814 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:27.054229 4814 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:27.023873 6932 ts_tablet_manager.cc:1428] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.048s user 0.007s sys 0.001s
I20260806 04:07:27.079804 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-2e4a7d4c7807476384d5ce509d4029f9 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:27.080307 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-2e4a7d4c7807476384d5ce509d4029f9 on tablet 2e4a7d4c7807476384d5ce509d4029f9 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:27.080955 6943 tablet_copy_client.cc:791] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 0 data blocks...
I20260806 04:07:27.081573 6943 tablet_copy_client.cc:655] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Starting download of 1 WAL segments...
I20260806 04:07:27.101246 4814 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:27.103245 4814 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:27.104451 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:27.104828 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:27.105063 5029 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:27.105428 4931 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:27.105746 4931 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:27.106488 4931 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:27.107914 4931 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:27.120472 5029 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:27.121837 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:27.131196 4931 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:27.130581 6851 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:27.139192 6851 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:27.140084 6851 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:27.142452 6942 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:27.144025 6942 tablet_bootstrap.cc:492] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:27.144698 6942 ts_tablet_manager.cc:1397] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.246s user 0.028s sys 0.023s
I20260806 04:07:27.148165 6942 raft_consensus.cc:348] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.149062 6942 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:27.149752 6942 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.151823 6942 ts_tablet_manager.cc:1428] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:07:27.159739 4831 tablet_copy_service.cc:342] P df2f1e7811114b8a995f6771fb6a6c8a: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-086f8af9433f44aa8aa4fccbfa90eb02 received from {username='slave'} at 127.0.0.1:53354
I20260806 04:07:27.160313 4831 tablet_copy_service.cc:433] P df2f1e7811114b8a995f6771fb6a6c8a: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-086f8af9433f44aa8aa4fccbfa90eb02 on tablet 086f8af9433f44aa8aa4fccbfa90eb02 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:27.167930 6943 tablet_copy_client.cc:531] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:07:27.202319 4668 catalog_manager.cc:5495] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
W20260806 04:07:27.227198 6968 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260806 04:07:27.243618 6938 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:27.248559 5862 raft_consensus.cc:1203] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:27.244587 6938 tablet_bootstrap.cc:492] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:27.268823 4674 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:27.272114 6938 ts_tablet_manager.cc:1397] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.362s user 0.027s sys 0.021s
I20260806 04:07:27.280575 6943 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap starting.
I20260806 04:07:27.296669 5862 raft_consensus.cc:1203] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20260806 04:07:27.320463 6938 raft_consensus.cc:348] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.321811 6938 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:27.322861 6938 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.325623 5862 raft_consensus.cc:1203] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:07:27.327436 6938 ts_tablet_manager.cc:1428] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.055s user 0.006s sys 0.002s
I20260806 04:07:27.329576 5049 tablet_copy_service.cc:342] P 99690415a7b3441fbdb5e943667556a4: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-997e4d9ff3ca465199a888e8af23ed7b received from {username='slave'} at 127.0.0.1:34146
I20260806 04:07:27.329995 5049 tablet_copy_service.cc:433] P 99690415a7b3441fbdb5e943667556a4: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-997e4d9ff3ca465199a888e8af23ed7b on tablet 997e4d9ff3ca465199a888e8af23ed7b with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:27.365423 4805 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:27.366866 6503 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:27.372396 4814 raft_consensus.cc:1261] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:27.381521 6957 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:07:27.405661 6943 log.cc:924] Time spent T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 04:07:27.422091 6943 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:07:27.455973 4813 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:07:27.460167 6943 tablet_bootstrap.cc:492] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Bootstrap complete.
I20260806 04:07:27.461508 6943 ts_tablet_manager.cc:1397] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent bootstrapping tablet: real 0.188s user 0.017s sys 0.013s
I20260806 04:07:27.459916 6503 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:27.471261 5862 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:27.481971 6879 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:27.490247 5029 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:07:27.496678 6665 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260806 04:07:27.515530 4931 raft_consensus.cc:1261] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:27.541168 6943 raft_consensus.cc:348] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.543147 6943 raft_consensus.cc:729] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Initialized, Role: LEARNER
I20260806 04:07:27.572227 6943 consensus_queue.cc:260] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }
I20260806 04:07:27.599152 6943 ts_tablet_manager.cc:1428] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent starting tablet: real 0.137s user 0.009s sys 0.005s
I20260806 04:07:27.607246 4944 tablet_copy_service.cc:342] P 22fe557f852d4415a96fd683f970cc24: Request end of tablet copy session d7c76b6dcd8945cd818b8107caf4a174-eaa370887c344934b64e0b3f59c6a3b0 received from {username='slave'} at 127.0.0.1:55924
I20260806 04:07:27.608039 4944 tablet_copy_service.cc:433] P 22fe557f852d4415a96fd683f970cc24: ending tablet copy session d7c76b6dcd8945cd818b8107caf4a174-eaa370887c344934b64e0b3f59c6a3b0 on tablet eaa370887c344934b64e0b3f59c6a3b0 with peer d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:27.609462 4927 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:27.617647 6970 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
W20260806 04:07:27.726395 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 89815.8us
W20260806 04:07:27.728745 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 93522us
I20260806 04:07:27.757184 6957 raft_consensus.cc:1050] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:27.757387 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:27.760928 6957 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:27.564145 6891 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
W20260806 04:07:27.899225 6976 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20260806 04:07:27.927551 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 289047us
W20260806 04:07:28.058951 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 84501.2us
W20260806 04:07:28.084846 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 344416us
I20260806 04:07:28.087828 5860 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEARNER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:07:28.088976 5029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:28.089656 5029 raft_consensus.cc:595] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:28.092175 7026 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:28.131018 5029 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:28.133536 4813 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260806 04:07:28.147084 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 82125.2us
W20260806 04:07:28.202822 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 72230.1us
I20260806 04:07:28.199954 7031 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
W20260806 04:07:28.208231 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 175754us
I20260806 04:07:28.219805 7018 raft_consensus.cc:1050] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
W20260806 04:07:28.249219 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 101381us
W20260806 04:07:28.257596 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 53886.5us
I20260806 04:07:28.267006 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:28.267632 4923 raft_consensus.cc:595] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:28.275692 4919 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260806 04:07:28.364812 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 83412.4us
I20260806 04:07:28.389519 5029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57d20919f4994e71841cccb9546444a3"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:28.390280 5029 raft_consensus.cc:595] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:28.389254 7018 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:28.433164 6991 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:28.445968 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 61909.3us
I20260806 04:07:28.470373 7031 raft_consensus.cc:979] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:28.471182 7031 raft_consensus.cc:1067] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
W20260806 04:07:28.480723 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 96550.1us
I20260806 04:07:28.485405 4930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:28.486073 4930 raft_consensus.cc:595] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:28.440366 4804 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:28.525018 4816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:28.525898 4816 raft_consensus.cc:595] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:28.487784 5860 raft_consensus.cc:1203] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:07:28.534678 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:07:28.544206 4927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:28.544907 4927 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:28.545280 4927 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:28.529889 7032 raft_consensus.cc:979] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
W20260806 04:07:28.545821 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 139727us
I20260806 04:07:28.557940 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:28.558907 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:07:28.557955 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 149375us
W20260806 04:07:28.569923 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 54630.2us
I20260806 04:07:28.568852 5867 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:28.507381 7014 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260806 04:07:28.590853 6665 raft_consensus.cc:979] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:28.591399 6665 raft_consensus.cc:1067] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:28.553349 6985 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260806 04:07:28.609439 4927 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:28.598963 7044 log.cc:924] Time spent T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260806 04:07:28.586222 7036 raft_consensus.cc:1067] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:28.615698 5860 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEARNER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:28.642354 4927 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:28.648154 7050 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 04:07:28.660483 4927 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:28.691291 6889 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 04:07:28.692670 7055 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 04:07:28.696800 6970 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260806 04:07:28.706321 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 137249us
I20260806 04:07:28.712388 5029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34042
I20260806 04:07:28.713181 5029 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:28.713606 5029 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:28.816859 4930 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:28.716539 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:28.981019 5032 raft_consensus.cc:3032] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:28.982506 5032 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:28.977837 7000 raft_consensus.cc:1050] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:28.978695 7040 raft_consensus.cc:1050] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:28.988389 4814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57d20919f4994e71841cccb9546444a3"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33564
I20260806 04:07:28.988940 4814 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:28.989261 4814 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:28.911813 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 61325.5us
I20260806 04:07:28.984180 5032 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:28.992074 5032 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:28.848145 5029 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.006902 5029 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:29.017409 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:29.018093 4804 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.021989 5026 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:29.034920 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:29.035622 4931 raft_consensus.cc:3032] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:29.035930 4931 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:29.036724 4931 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.037899 4931 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.030694 7005 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:29.064491 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:29.114220 5860 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.130628 5035 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:29.019814 4920 raft_consensus.cc:2932] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:29.098438 7005 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.139060 7005 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:29.130059 4931 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:29.151335 4674 catalog_manager.cc:5495] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 reported cstate change: config changed from index 3 to 4, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 04:07:29.183507 7062 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260806 04:07:29.217155 7007 raft_consensus.cc:1050] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: attempting to promote NON_VOTER d7c76b6dcd8945cd818b8107caf4a174 to VOTER
I20260806 04:07:29.213639 6950 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:29.227694 4814 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.237890 7007 consensus_queue.cc:237] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:29.245476 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 187069us
I20260806 04:07:29.247167 4804 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:29.250888 4814 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:29.250910 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 214941us
I20260806 04:07:29.253901 5032 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:29.268615 4804 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:29.288890 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:29.297526 4882 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:29.305388 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:29.289602 6851 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260806 04:07:29.302955 5860 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:29.312448 5866 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:29.315063 4995 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:29.300854 6877 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:29.314114 7019 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:29.320454 6877 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.321290 6991 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:29.321095 7019 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.322211 6991 raft_consensus.cc:686] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:29.324698 7002 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:29.323612 6991 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.325189 7019 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:29.325614 7002 raft_consensus.cc:686] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:29.327520 4926 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:29.342927 7018 raft_consensus.cc:979] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:29.343669 7018 raft_consensus.cc:1067] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:29.345245 6877 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:29.340495 5866 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:29.347877 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:29.346534 7002 consensus_queue.cc:237] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.349134 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:29.350071 5859 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.355455 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:29.359402 6665 raft_consensus.cc:979] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:29.359871 6665 raft_consensus.cc:1067] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:29.366827 4804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:29.369786 4804 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:29.370319 4804 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.386245 4924 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45408
I20260806 04:07:29.387073 4924 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:29.387502 4924 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:29.422283 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:29.421725 4924 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.356072 7082 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260806 04:07:29.433799 4924 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:29.437090 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:29.437929 4813 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:29.438298 4813 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:29.443630 5032 raft_consensus.cc:3032] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:29.444133 5032 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:29.445295 5032 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.439302 4813 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.447700 5030 raft_consensus.cc:2932] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:29.452399 4813 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:29.455983 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:29.456831 5866 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:29.452714 4804 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.466217 4804 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:29.467514 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:29.468564 4926 raft_consensus.cc:3032] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:29.468962 4926 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:29.477617 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:29.440791 7000 consensus_queue.cc:237] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.470073 4926 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.508226 5867 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:29.521822 4926 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.535393 4926 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:29.531949 7007 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260806 04:07:29.556130 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:29.590631 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 47440us
I20260806 04:07:29.593744 5032 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.602123 5866 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:29.611866 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:29.612946 4813 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:29.611817 4879 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:29.634017 5032 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:29.646695 5859 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:29.649480 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:29.651621 4814 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:29.651736 4992 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:29.653014 5867 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:29.653412 5030 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:29.654304 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:29.655061 5034 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:29.626634 6851 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:29.641911 6956 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260806 04:07:29.661907 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:29.671599 4770 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:29.673223 4768 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:29.673228 6851 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:29.657886 6957 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:29.696041 7024 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:29.675678 6970 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:29.705570 5866 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 LEARNER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:29.702069 7024 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:29.735888 5034 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:29.790791 7056 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:29.794569 7056 raft_consensus.cc:686] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:29.845853 7040 raft_consensus.cc:1054] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Unable to promote non-voter d7c76b6dcd8945cd818b8107caf4a174: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: NON_VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: true } }.
Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.777367 7074 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.152s
I20260806 04:07:29.850178 7048 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
W20260806 04:07:29.864136 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 185403us
W20260806 04:07:29.883559 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 172178us
I20260806 04:07:29.782043 7024 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.884709 7020 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:29.904994 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:29.905791 4923 raft_consensus.cc:595] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:29.880718 6851 raft_consensus.cc:686] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:29.796430 7056 consensus_queue.cc:237] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:29.927129 6851 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.940599 5035 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57d20919f4994e71841cccb9546444a3"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:29.941578 5035 raft_consensus.cc:595] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:29.942022 5035 raft_consensus.cc:603] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:29.944281 7106 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:29.950591 7007 raft_consensus.cc:979] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:29.955152 7007 raft_consensus.cc:1067] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:29.946102 7106 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:29.959489 4816 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:29.966231 4925 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:29.972411 4814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:29.973094 4814 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:29.973476 4814 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:29.975365 5030 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:29.975953 4816 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:29.977330 4816 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "874b55144315439cb842845d56ccb153"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:29.977989 4816 raft_consensus.cc:595] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:29.980283 5035 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:07:29.983008 7000 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260806 04:07:29.997269 7000 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:29.983093 6957 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 04:07:30.019918 7106 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:30.022760 5859 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:30.027338 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:30.039467 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:30.026125 7002 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:30.043337 7031 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:30.074604 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:30.049289 4770 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:30.082798 5034 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:30.083591 5034 raft_consensus.cc:595] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:30.086642 4814 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:30.093833 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51818.3us
I20260806 04:07:30.096066 4814 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:30.062002 6879 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260806 04:07:30.079210 7014 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:30.123368 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46150.5us
I20260806 04:07:30.153334 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:30.159910 5035 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:30.129038 4919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
W20260806 04:07:30.169970 7114 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260806 04:07:30.190893 4813 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:30.211711 4919 raft_consensus.cc:595] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:30.221333 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:07:30.207190 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:30.253654 4925 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:30.271559 6851 raft_consensus.cc:979] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:30.319021 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:30.315519 7102 raft_consensus.cc:1067] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:30.311161 4925 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:30.580209 7116 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260806 04:07:30.876060 5035 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.690s user 0.008s sys 0.011s
I20260806 04:07:30.590313 4925 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:30.193951 7080 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:30.851203 4814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.009s sys 0.003s
W20260806 04:07:30.885064 4814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s user 0.009s sys 0.003s
I20260806 04:07:30.880719 5035 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:30.880818 4813 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:30.889513 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:07:30.928210 7120 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.052s user 0.002s sys 0.000s
I20260806 04:07:30.959757 4925 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:31.090580 7118 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.554s user 0.001s sys 0.010s
I20260806 04:07:31.105769 6956 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:31.111605 6956 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.091146 7099 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.141198 7132 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:31.141880 7132 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.150141 5028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "874b55144315439cb842845d56ccb153"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:31.161846 5028 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:31.162405 5028 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:31.150125 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 128939us
I20260806 04:07:31.157819 5867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 412) took 1088 ms. Trace:
I20260806 04:07:31.172135 5862 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:31.167579 5867 rpcz_store.cc:276] 0806 04:07:30.064527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:30.064731 (+ 204us) service_pool.cc:227] Handling call
0806 04:07:30.065245 (+ 514us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:30.065664 (+ 419us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:30.065730 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:30.066298 (+ 568us) log.cc:842] Serialized 26 byte log entry
0806 04:07:30.106637 (+ 40339us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:30.106878 (+ 241us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:30.106947 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:31.152431 (+1045484us) raft_consensus.cc:1671] finished
0806 04:07:31.152479 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:31.152640 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":4212,"thread_start_us":39874,"threads_started":1,"wal-append.queue_time_us":246013}
I20260806 04:07:31.197425 7031 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:31.198189 7031 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.201709 7031 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:31.129309 7099 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.236860 5028 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.239241 5867 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:31.248176 5028 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:31.264883 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 166019us
I20260806 04:07:31.212886 7040 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260806 04:07:31.273604 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:31.345547 5866 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
W20260806 04:07:31.347817 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 71687us
I20260806 04:07:31.350343 6879 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:31.352093 7132 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:31.349761 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 74676.7us
I20260806 04:07:31.356897 7099 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:31.279284 6956 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:31.367450 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.367945 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:31.369848 4995 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:31.369895 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:31.373675 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:31.372460 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.345080 7131 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:31.375715 7131 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.379395 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:31.380167 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.379235 5859 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:31.407423 7007 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:31.411450 7034 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:31.414461 7003 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:31.415256 7003 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.425350 7015 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:31.427600 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54554.7us
I20260806 04:07:31.427486 7141 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260806 04:07:31.435142 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.435849 4813 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:31.412312 7034 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.438277 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:31.462873 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 84658us
I20260806 04:07:31.440667 5816 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:31.465008 7003 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:31.408228 7007 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.373829 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:31.470042 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:31.426110 7015 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:31.489444 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 115715us
I20260806 04:07:31.378484 7131 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:31.493458 5035 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:31.465334 6984 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:31.498361 6893 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.499819 6956 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:31.500088 7138 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.500845 5035 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:31.500993 7099 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:31.503185 7141 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.504079 7031 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.487298 7137 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:31.468825 6957 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:31.520107 4674 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46902 (request call id 281) took 2852 ms. Trace:
I20260806 04:07:31.522838 6851 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.524161 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.538885 4669 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46886 (request call id 286) took 2836 ms. Trace:
I20260806 04:07:31.534485 4814 raft_consensus.cc:3032] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:31.542799 4814 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:31.536197 4674 rpcz_store.cc:276] 0806 04:07:28.660548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:28.660713 (+ 165us) service_pool.cc:227] Handling call
0806 04:07:31.513129 (+2852416us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:29.360227 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:29.375086 (+ 14859us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:29.375155 (+ 69us) write_op.cc:435] Acquired schema lock
0806 04:07:29.375208 (+ 53us) tablet.cc:660] Decoding operations
0806 04:07:29.375786 (+ 578us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:29.376039 (+ 253us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:29.376067 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:07:29.376335 (+ 268us) tablet.cc:699] Row locks acquired
0806 04:07:29.376356 (+ 21us) write_op.cc:260] PREPARE: finished
0806 04:07:29.376613 (+ 257us) write_op.cc:270] Start()
0806 04:07:29.376849 (+ 236us) write_op.cc:276] Timestamp: P: 1785989249376571 usec, L: 0
0806 04:07:29.376874 (+ 25us) op_driver.cc:331] REPLICATION: starting
0806 04:07:29.378441 (+ 1567us) log.cc:842] Serialized 595 byte log entry
0806 04:07:29.934604 (+556163us) op_driver.cc:444] REPLICATION: finished
0806 04:07:30.878025 (+943421us) write_op.cc:301] APPLY: starting
0806 04:07:30.881020 (+ 2995us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:30.881186 (+ 166us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:30.908122 (+ 26936us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:30.922775 (+ 14653us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:30.958724 (+ 35949us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:07:30.971815 (+ 13091us) write_op.cc:312] APPLY: finished
0806 04:07:30.987924 (+ 16109us) log.cc:842] Serialized 23 byte log entry
0806 04:07:31.129793 (+141869us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:31.129914 (+ 121us) write_op.cc:454] Released schema lock
0806 04:07:31.129977 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:31.149728 (+ 19751us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:31.151615 (+ 1887us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":240408,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":918110,"apply.run_cpu_time_us":13879,"apply.run_wall_time_us":281148,"num_ops":1,"prepare.queue_time_us":195506,"prepare.run_cpu_time_us":120570,"prepare.run_wall_time_us":265919,"raft.queue_time_us":240031,"raft.run_cpu_time_us":11585,"raft.run_wall_time_us":489277,"replication_time_us":546357,"thread_start_us":1094136,"threads_started":5,"wal-append.queue_time_us":567312,"wal-append.run_cpu_time_us":35193,"wal-append.run_wall_time_us":1227201}]]}
I20260806 04:07:31.539430 4669 rpcz_store.cc:276] 0806 04:07:28.702164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:28.702366 (+ 202us) service_pool.cc:227] Handling call
0806 04:07:31.538804 (+2836438us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":280039,"reported_tablets":1}
I20260806 04:07:31.520324 4925 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:31.573100 7039 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:31.521247 6984 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:31.570520 4814 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.530436 7099 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.571835 6851 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.575317 4814 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:31.541150 7140 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:31.537173 4920 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:31.544747 7137 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:31.525573 6956 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.597702 6851 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:31.576336 6984 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:31.536167 7031 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.613866 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:31.587797 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1354) took 1451 ms. Trace:
I20260806 04:07:31.501837 7146 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:31.520975 6893 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:31.644841 7118 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.113s user 0.002s sys 0.000s
I20260806 04:07:31.628345 4814 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
W20260806 04:07:31.636670 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 57258.9us
I20260806 04:07:31.600823 7040 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:31.634687 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:31.582448 7140 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.554003 6957 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.652813 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.652456 4804 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:07:31.659164 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 81441.8us
I20260806 04:07:31.659817 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:31.658010 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 58376.5us
I20260806 04:07:31.542181 7141 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.639111 6879 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:31.530148 7138 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.668851 5863 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:31.675339 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:31.738571 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:31.744325 4923 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:31.749521 6955 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.752240 4923 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
W20260806 04:07:31.752684 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 81853.4us
I20260806 04:07:31.749547 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:31.631822 7014 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260806 04:07:31.663970 7015 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:31.757814 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:31.673204 6889 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:31.632398 4925 rpcz_store.cc:276] 0806 04:07:30.127844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:30.144163 (+ 16319us) service_pool.cc:227] Handling call
0806 04:07:31.577298 (+1433135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816}
I20260806 04:07:31.658569 7040 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:31.803932 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 144158us
I20260806 04:07:31.750298 6955 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.774394 7007 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:31.805119 6889 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260806 04:07:31.885967 7165 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260806 04:07:31.678983 6879 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:31.809319 7140 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:32.118078 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:31.886685 7134 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.860131 7168 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:31.751273 7137 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:31.796681 7020 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:31.804548 7149 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.122397 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:32.123528 5859 consensus_meta.cc:287] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.742s user 0.009s sys 0.009s
I20260806 04:07:32.137529 7166 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:32.137593 7162 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.376s user 0.003s sys 0.005s
I20260806 04:07:32.166167 7141 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:32.138535 7099 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:32.181800 7167 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.183013 7172 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.150678 7031 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.134197 6893 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:32.164583 6956 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.176973 6957 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:32.174517 7007 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:32.227129 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:32.225411 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:32.224228 6955 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.233359 7040 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:32.233953 6879 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:32.229988 6891 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.206120 7031 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:32.162833 7020 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.174029 7168 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.241108 7031 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.241523 7039 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.241984 7039 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:32.238232 6891 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.234320 7028 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.248176 7031 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.248490 6891 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.168655 7134 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.196334 7034 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:32.225754 7007 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.185488 7149 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.190227 7166 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.208794 7167 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.270850 7138 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.211483 7172 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:32.316339 7146 consensus_meta.cc:287] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.515s user 0.010s sys 0.004s
W20260806 04:07:32.313809 6984 consensus_meta.cc:287] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.695s user 0.010s sys 0.004s
I20260806 04:07:32.329255 7168 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.333168 7167 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.320166 7134 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.339424 6984 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.318748 7149 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:32.346498 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 180915us
I20260806 04:07:32.266072 7028 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.338887 7020 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:32.348286 7166 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.435451 5028 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:32.437494 7155 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.841s user 0.009s sys 0.003s
I20260806 04:07:32.442580 7028 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:32.443678 7174 log.cc:924] Time spent T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260806 04:07:32.445325 4996 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:32.445502 7167 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.446115 7167 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.450019 7182 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:32.450812 7182 raft_consensus.cc:686] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:32.454975 5032 raft_consensus.cc:2932] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:32.481130 7172 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.553167 7166 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.553946 7166 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.557989 7166 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.571318 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1395) took 1094 ms. Trace:
I20260806 04:07:32.571961 5028 rpcz_store.cc:276] 0806 04:07:31.476584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:31.476704 (+ 120us) service_pool.cc:227] Handling call
0806 04:07:31.476937 (+ 233us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:32.435885 (+958948us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:32.570979 (+135094us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b54000bca54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:32.571182 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2203898624}
I20260806 04:07:32.576192 7167 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.452082 7182 consensus_queue.cc:237] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.578410 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1405) took 1079 ms. Trace:
I20260806 04:07:32.578922 5034 rpcz_store.cc:276] 0806 04:07:31.498909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:31.499020 (+ 111us) service_pool.cc:227] Handling call
0806 04:07:31.499314 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:31.674455 (+175141us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:31.674475 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:31.674499 (+ 24us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:31.674515 (+ 16us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:32.578163 (+903648us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54000bcf54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:32.578210 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:32.578348 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402389888}
I20260806 04:07:32.595497 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:32.610786 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:32.613567 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 461898us
I20260806 04:07:32.615108 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:32.619132 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 375506us
I20260806 04:07:32.619860 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:32.633910 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:32.635393 6665 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:32.642355 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:32.636044 6665 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:32.644052 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:32.646412 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:32.650213 6665 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:32.650996 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:32.651237 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:32.651448 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 449290us
I20260806 04:07:32.664332 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:32.671523 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:32.769819 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:32.767515 4770 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.007s sys 0.033s
W20260806 04:07:32.770642 4770 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.008s sys 0.033s
I20260806 04:07:32.785019 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:32.797969 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:32.761922 7029 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:32.674789 7039 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:32.813691 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46951.9us
I20260806 04:07:32.778851 7073 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:32.870875 5860 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:32.841315 7167 raft_consensus.cc:979] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: : Instructing follower 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:32.903327 7107 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:32.904114 7107 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.827488 7029 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.889931 7202 raft_consensus.cc:1067] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Signalling peer 22fe557f852d4415a96fd683f970cc24 to start an election
I20260806 04:07:32.930302 7080 raft_consensus.cc:979] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:32.930968 7080 raft_consensus.cc:1067] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:32.922068 7193 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:32.951129 7193 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.953769 7193 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:32.965114 7039 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:32.857888 7073 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:32.946250 6984 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.031749 7073 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:33.033309 7210 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260806 04:07:33.020920 7107 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:33.051144 7203 log.cc:924] Time spent T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
W20260806 04:07:33.093029 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 260518us
I20260806 04:07:33.100976 4992 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:33.108057 4992 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:33.120571 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.123286 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:33.120877 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.124830 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:33.126032 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:33.127077 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:33.128266 5862 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:33.128852 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.130033 4816 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:33.130707 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:33.130982 5859 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.921s user 0.008s sys 0.040s
W20260806 04:07:33.131582 5859 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s user 0.008s sys 0.040s
I20260806 04:07:33.123450 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.138736 7202 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:33.139230 6889 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.140679 6889 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.141877 5032 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:33.145489 7015 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.146334 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:33.146520 7015 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.147652 6851 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:33.138779 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:33.142500 7202 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.168453 7103 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.168934 7080 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.169341 7103 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.171096 7140 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.171260 7144 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.171468 6851 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.172073 7140 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.172302 7144 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:33.172855 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 340262us
I20260806 04:07:33.174232 6851 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.175057 7190 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.175153 6891 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:33.176074 7190 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.176692 7195 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.170274 5028 raft_consensus.cc:2932] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:33.177562 7195 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.179077 7198 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.180146 7000 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.180109 7198 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.181354 7000 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.183084 7184 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.184124 7007 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.183995 7184 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.180122 7073 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.186955 7018 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.183146 7199 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.187372 7073 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.187889 7018 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.188278 7199 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.189615 6970 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.179199 7039 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.168690 4672 catalog_manager.cc:5495] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 2 to 3, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 3 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:33.190315 6970 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.194109 7137 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.167140 7182 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.194525 7000 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.191349 4805 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:33.191900 6851 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.190353 7018 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:33.160832 4995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.017s sys 0.020s
I20260806 04:07:33.160588 7030 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.167033 7040 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.206527 7040 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.207450 7103 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.208243 7186 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.209075 7040 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.197984 7039 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.211261 5036 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
W20260806 04:07:33.212829 4995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.969s user 0.019s sys 0.023s
I20260806 04:07:33.212917 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.213744 7039 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.214012 4810 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:33.160429 7167 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.215070 7167 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.220817 4668 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56998 (request call id 140) took 4094 ms. Trace:
I20260806 04:07:33.221231 4668 rpcz_store.cc:276] 0806 04:07:29.126552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:29.126722 (+ 170us) service_pool.cc:227] Handling call
0806 04:07:33.220522 (+4093800us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19165,"reported_tablets":4}
I20260806 04:07:33.224413 7029 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:33.165963 7139 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:33.154595 7002 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.225100 4672 catalog_manager.cc:5495] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: config changed from index 4 to 5, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:33.192225 7140 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.234057 7030 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.237133 7030 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.208992 7186 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.170197 7080 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.231462 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.245769 6851 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.245252 4813 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:33.230459 7190 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.248301 7199 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.245819 4672 catalog_manager.cc:5495] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:33.249114 7029 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.267635 7193 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.249917 6889 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.268462 7193 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.226052 7139 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.257928 7144 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.261233 7198 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.270898 7128 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.244482 7137 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.284687 6851 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.245043 7202 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.255364 7002 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.289565 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.278841 7182 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.281713 7184 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.283217 7080 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.279038 7190 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:33.475970 7001 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:33.363903 7029 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.481048 7139 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.480928 7190 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.299171 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:33.388955 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 97477.4us
I20260806 04:07:33.300419 6851 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.278699 7199 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.300859 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:33.485721 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:33.489578 7211 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260806 04:07:33.441439 7128 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.459729 7195 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:33.306429 4651 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leader transfer in progress [suppressed 3 similar messages]
W20260806 04:07:33.491073 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 128751us
I20260806 04:07:33.490808 6851 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.492803 7002 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.494797 7029 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:33.497740 7190 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.499792 7202 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:33.420389 7193 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.419358 7003 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:33.485145 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 150) took 2133 ms. Trace:
I20260806 04:07:33.575999 5859 rpcz_store.cc:276] 0806 04:07:31.260229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:31.263715 (+ 3486us) service_pool.cc:227] Handling call
0806 04:07:31.378724 (+115009us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:31.379047 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:07:32.179990 (+800943us) log.cc:842] Serialized 14 byte log entry
0806 04:07:32.626188 (+446198us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d6a145fe 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb
0806 04:07:33.262794 (+636606us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:33.276323 (+ 13529us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:33.350359 (+ 74036us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:33.356113 (+ 5754us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:33.393667 (+ 37554us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43257,"raft.queue_time_us":51923,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":279,"spinlock_wait_cycles":7335296,"thread_start_us":997524,"threads_started":2,"wal-append.queue_time_us":1098970}
I20260806 04:07:33.502924 6970 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.504168 7167 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:33.514145 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.515435 7137 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.515314 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.516325 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.517412 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.442093 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:33.310313 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:33.378387 7015 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.551637 7128 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.335623 7144 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:07:33.569993 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 462222us
I20260806 04:07:33.364141 7073 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.308892 7186 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:33.400352 7218 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:33.812351 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 301350us
I20260806 04:07:33.723264 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.814970 6851 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:33.819692 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 314843us
I20260806 04:07:33.814179 5863 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:33.808923 7219 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:33.824196 7219 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:33.825330 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 343305us
I20260806 04:07:33.827674 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.641247 7199 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.835006 7080 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:33.835214 7218 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.837257 7199 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.841050 7144 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.843422 7144 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.684702 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.844408 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:33.956880 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.965448 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:33.739238 6970 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.149083 4931 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:34.149637 6970 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.957559 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.152628 5860 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:33.925163 7186 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:34.154202 4919 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:34.154274 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.154567 7186 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:33.835542 7209 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.282s user 0.000s sys 0.000s
I20260806 04:07:33.962287 4809 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:34.157047 7030 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.157402 7184 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.157521 4879 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:34.157809 7030 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.158241 7184 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:33.885645 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:33.856431 7137 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:34.160234 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.156394 7144 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.162880 7144 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.160315 7030 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.164669 7007 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.165988 7007 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.166354 7144 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.084551 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:34.045516 4651 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 739250us
I20260806 04:07:33.809428 7099 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.160319 7184 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:33.738773 7103 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:34.183264 7103 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.191562 7007 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.191764 7103 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.193377 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.194900 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.194921 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:34.196179 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:34.160558 7137 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.197618 4809 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
W20260806 04:07:34.198158 7225 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260806 04:07:33.960728 7219 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:34.095582 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 413981us
I20260806 04:07:34.103535 7193 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.171687 7190 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.171020 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.204372 7007 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.172206 7182 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.177727 7214 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.205384 7007 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.205525 7214 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.206322 7193 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.178772 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.180145 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.205024 5857 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:34.208902 7193 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.209817 7007 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.210475 7099 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.196820 7103 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:33.991966 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.201185 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.203832 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.204519 7190 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.173074 7218 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:34.212582 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:34.213732 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 369704us
W20260806 04:07:34.214802 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 447730us
I20260806 04:07:34.217804 7031 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.217963 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.218431 7031 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.219156 7190 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.219409 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.220321 5034 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:34.218116 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:33.592962 7003 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:33.845521 7080 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 04:07:34.250025 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 421027us
I20260806 04:07:34.249862 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1697) took 1072 ms. Trace:
I20260806 04:07:34.251945 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.266492 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.267030 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.267683 5030 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:34.268178 7144 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.268816 7144 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.269171 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:34.269452 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.270114 7002 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.270756 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.270985 7144 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.271121 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.271564 7144 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:34.272132 7144 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.272365 7184 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:34.275530 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.275728 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.276221 5036 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:34.275990 7184 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.224800 6970 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.276698 7218 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.277464 7144 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.277866 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.278002 7184 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.272320 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.272171 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.215389 7103 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.216173 7219 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.283789 4926 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:34.276314 5026 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.277215 7218 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.216935 7167 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.279210 5035 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.261901 4879 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:34.281699 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 48531us
I20260806 04:07:34.270787 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.225034 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.286643 5037 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:34.287197 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.273095 7002 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.288128 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.288506 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.284209 7219 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.288874 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.251649 7031 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.289710 4992 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.284441 7137 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.312484 4928 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:34.313963 7184 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.316772 7001 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.316661 7167 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.274361 7080 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:34.264611 5028 rpcz_store.cc:276] 0806 04:07:33.168160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:33.168865 (+ 705us) service_pool.cc:227] Handling call
0806 04:07:33.169357 (+ 492us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:33.169924 (+ 567us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:07:33.517054 (+347130us) log.cc:842] Serialized 14 byte log entry
0806 04:07:34.232352 (+715298us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:34.232399 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:34.232439 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:34.232528 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:34.232677 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":311809,"raft.queue_time_us":345060,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":10081,"thread_start_us":387685,"threads_started":1,"wal-append.queue_time_us":730670}
I20260806 04:07:34.314770 4924 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:34.317490 5030 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:34.399402 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.283991 7103 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.317634 5034 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:34.404044 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 124231us
I20260806 04:07:34.424517 5035 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
W20260806 04:07:34.405647 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 86724.7us
W20260806 04:07:34.428408 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 138799us
I20260806 04:07:34.405872 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.429273 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.429332 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.430513 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.430624 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.429548 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.434808 7144 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:34.435062 4919 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:34.435318 4876 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.437721 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.438429 5867 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:34.432987 4930 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:34.436997 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.361651 7186 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.405159 7214 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.425077 7219 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.440121 4992 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.423627 7218 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.418689 7184 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.423542 7099 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.445647 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.447154 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.448972 7144 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:34.445385 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.434906 7001 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.453037 7001 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.448688 7003 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.439726 7031 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:34.455353 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.486792 7190 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.489672 7080 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.482587 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.491297 7080 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:34.491386 7002 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.491019 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.492599 7190 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.494439 4920 raft_consensus.cc:2379] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:34.484582 5867 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:34.494745 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.446278 4931 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:34.455787 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:34.498378 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.500752 7003 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.501420 7003 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.503536 7003 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.478924 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.504662 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:34.505545 7099 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:34.480906 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 52978.4us
I20260806 04:07:34.511003 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.506886 7099 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.511850 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:34.512372 7099 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.465425 7031 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:34.512524 5867 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.508740 7184 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.541193 4813 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:34.543136 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.546108 4768 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:34.552834 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:34.553758 7218 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:34.554162 7218 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.554553 7218 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.550845 6970 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.555424 7003 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.557193 7003 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.557564 7003 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:34.555238 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 112858us
I20260806 04:07:34.563549 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.589051 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.589877 4920 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:34.590286 4770 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.591178 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.591547 4770 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:34.591775 7214 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:34.592315 7137 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:34.592657 7137 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.593008 7137 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:34.593729 5812 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:34.556713 7219 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.594519 7205 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.594944 7205 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.595310 7205 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.555814 7167 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.603142 6851 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:34.602042 7214 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:34.607059 6851 raft_consensus.cc:3032] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:34.607192 7204 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.607430 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.607915 7204 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.608647 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.609813 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.610102 7204 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.571744 4809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:34.611688 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.561986 7186 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.611152 4809 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:34.607445 6851 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:34.614485 4809 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.611109 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.616469 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:34.564172 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.567770 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.613588 7202 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.619830 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.620096 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.620512 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.621537 4768 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.622673 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.622964 7219 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.620575 6851 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.555653 6970 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.623402 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.654078 4804 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.655125 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.615672 7214 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:34.624051 7198 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:34.656807 7198 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.623957 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.658756 7214 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:34.623700 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.658002 4809 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.659289 5866 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:34.661832 4814 raft_consensus.cc:2379] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:34.612564 7204 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:34.662981 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.666157 7199 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:34.664515 4995 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:34.668147 6851 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.664763 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.670112 4809 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.614395 7186 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.670562 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.670836 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.659546 7198 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.672188 4816 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.674113 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.671838 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.672089 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.673543 7186 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.684599 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.682891 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.687549 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.688827 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.690893 7031 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.691464 7031 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.692808 5028 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.693641 7031 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.694517 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.694774 7182 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:34.695200 7182 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:34.695624 7182 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.695876 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.696031 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.695603 5867 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.695645 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.695082 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.698487 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.699141 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.699476 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.621634 7003 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.700186 4879 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:34.699306 4770 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:34.689558 4808 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.701175 4817 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.682299 7249 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:34.706251 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.706306 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.707276 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.721064 4928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:45410
I20260806 04:07:34.711951 7080 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:34.715346 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.669188 7214 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.674467 7202 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.719498 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.726634 4928 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:34.728668 7214 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.729408 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.730719 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.731441 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.734277 4928 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:34.733244 4813 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.743326 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:34.745680 4817 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.746562 4808 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:34.719270 7219 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:34.748797 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 77458.7us
I20260806 04:07:34.685402 7204 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:34.750468 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 65551.6us
I20260806 04:07:34.751789 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.697158 7103 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.724639 7199 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:34.726573 7249 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.751258 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:34.785356 5857 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:34.944209 7186 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:34.945660 4818 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:34.744784 7080 raft_consensus.cc:686] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:34.946893 7080 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.780125 7205 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.938494 7218 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.961973 7249 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:34.962131 7204 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:34.962504 7218 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.962942 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:34.962836 7257 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:34.964097 5862 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:34.964180 7186 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.951982 4928 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:34.895623 7239 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260806 04:07:34.949783 7205 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.839507 6970 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:34.975641 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 127627us
I20260806 04:07:34.977558 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.978324 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.973538 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:34.966965 7202 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:34.964818 7003 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.984434 7103 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:34.984937 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.985569 5029 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:34.781517 7099 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.989666 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.939499 7182 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:34.990319 5036 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:34.995016 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:34.998948 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:34.999713 4928 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.004663 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:35.005105 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:35.005264 5867 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.006531 7099 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.008104 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.008801 7137 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.009212 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.009771 5026 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:34.968967 7199 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.010838 7249 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:35.012210 7202 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:35.012259 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.003809 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.025137 7199 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.025624 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:35.025920 5026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8570417a88db412294c21c9108061c30"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:35.026336 4919 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.026607 5026 raft_consensus.cc:595] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:35.027110 7253 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:35.027673 7253 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.029367 4808 raft_consensus.cc:1226] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.029628 7253 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.030344 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.031603 7030 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.031941 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.031612 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.032358 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.032753 4810 raft_consensus.cc:2396] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:07:35.033272 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.033485 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.034216 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.034454 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.034596 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.035228 5038 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:35.035530 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.036118 7182 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.036424 5034 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.037352 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.033679 4817 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.023571 5867 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:35.030387 4876 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:35.039259 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.042124 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:35.042732 5867 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:34.998375 7198 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.045311 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.053102 4876 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:35.054061 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.047019 7030 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:35.055713 7030 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.055833 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.056182 7030 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.057075 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.015695 7202 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.057375 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.058104 5028 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:35.058344 7202 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:35.045679 7184 consensus_meta.cc:287] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.518s user 0.007s sys 0.007s
I20260806 04:07:35.123993 4805 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:35.135814 4931 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.144718 5034 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:35.135905 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.066375 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.048058 7137 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.112411 4816 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
W20260806 04:07:35.120241 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 366940us
I20260806 04:07:35.066799 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.048234 5854 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.092666 5036 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:35.215833 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.214699 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.057318 7002 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:35.051786 4668 catalog_manager.cc:5495] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:35.217325 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.143950 7202 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:35.206861 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 153826us
I20260806 04:07:35.175264 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.207073 5032 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:35.186424 4817 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:35.136154 4927 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:35.092962 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.210484 5867 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:35.138170 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.129350 4672 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46900 (request call id 280) took 4955 ms. Trace:
I20260806 04:07:35.067165 5035 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:35.214154 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.155383 7184 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:35.119985 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 115960us
I20260806 04:07:35.055117 7144 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.217636 7002 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.249480 4927 raft_consensus.cc:595] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.220965 7198 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.221451 4668 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:35.258852 7002 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:35.222565 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.153374 5857 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 04:07:35.223343 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.105746 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.224651 4808 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:35.247763 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.248260 4924 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.247822 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.236415 4804 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:35.266616 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 160697us
I20260806 04:07:35.249784 5038 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:35.249009 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.205744 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:35.254580 4995 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:35.268702 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.266001 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.250809 4876 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:35.259163 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.262221 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.261726 4668 catalog_manager.cc:5495] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: config changed from index 2 to 3, d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:35.267752 4810 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:35.271765 4668 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:35.273257 4876 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:35.273615 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 67958.9us
I20260806 04:07:35.273880 5034 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:35.274698 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.276335 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.279441 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.279907 5032 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:35.274257 4816 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:35.282681 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.283080 5816 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:35.283313 7184 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.284763 7218 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:35.287061 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.287552 5029 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:35.275404 6891 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:35.289651 4768 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.285255 7218 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:35.273545 7270 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:35.298569 5863 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.299026 4924 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:35.301852 4924 raft_consensus.cc:595] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.295554 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:35.288960 6891 raft_consensus.cc:686] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.298817 4814 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:35.306581 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:35.307003 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:35.277582 7198 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.309872 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:35.310611 4923 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.293561 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:35.299199 7268 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.313041 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.313995 4924 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:35.309216 7182 consensus_queue.cc:1046] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260806 04:07:35.317006 7268 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:35.313205 7198 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:35.317706 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.317598 7268 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.318137 7198 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.318346 7257 consensus_queue.cc:1046] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:35.319453 7257 raft_consensus.cc:3032] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:35.319793 7257 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.319895 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.319692 6891 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.320611 4925 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:35.320717 7257 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.322057 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:35.321924 7253 consensus_queue.cc:1046] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:35.304104 7270 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.322678 4928 raft_consensus.cc:3032] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:35.322919 7270 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.323035 4928 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:35.323113 7253 raft_consensus.cc:3032] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:35.332043 7253 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:35.332506 7030 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.333030 7253 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.317880 7182 raft_consensus.cc:3032] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:35.322245 7257 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.293932 7184 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.323913 4928 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.317035 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:35.348430 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 75318.6us
I20260806 04:07:35.349066 4930 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.326069 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.331274 7198 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.335860 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.335893 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.334293 7253 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.351101 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.340713 7268 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.340822 7190 raft_consensus.cc:979] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:35.336515 7182 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.341670 7218 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.342379 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.342438 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:35.365577 4931 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.248746 4672 rpcz_store.cc:276] 0806 04:07:30.167238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:30.167401 (+ 163us) service_pool.cc:227] Handling call
0806 04:07:30.893251 (+725850us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54001c1954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23df8b4d6c 00007f23df9aec3e 00007f23ddcc0fb4 00007f23ddcc0f42 00007f23ddcc0e71 00007f23ddcc0dec 00007f23ddcc00a2 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:35.122393 (+4229142us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:34.177646 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:34.178512 (+ 866us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:34.178570 (+ 58us) write_op.cc:435] Acquired schema lock
0806 04:07:34.178599 (+ 29us) tablet.cc:660] Decoding operations
0806 04:07:34.179337 (+ 738us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:34.179451 (+ 114us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:34.179512 (+ 61us) tablet.cc:683] Acquiring locks for 3 operations
0806 04:07:34.180137 (+ 625us) tablet.cc:699] Row locks acquired
0806 04:07:34.180178 (+ 41us) write_op.cc:260] PREPARE: finished
0806 04:07:34.180359 (+ 181us) write_op.cc:270] Start()
0806 04:07:34.180523 (+ 164us) write_op.cc:276] Timestamp: P: 1785989254180297 usec, L: 0
0806 04:07:34.180553 (+ 30us) op_driver.cc:331] REPLICATION: starting
0806 04:07:34.181350 (+ 797us) log.cc:842] Serialized 1360 byte log entry
0806 04:07:34.589319 (+407969us) op_driver.cc:444] REPLICATION: finished
0806 04:07:34.676546 (+ 87227us) write_op.cc:301] APPLY: starting
0806 04:07:34.676688 (+ 142us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:34.676838 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:34.676981 (+ 143us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:34.677549 (+ 568us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:34.677828 (+ 279us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 04:07:34.681965 (+ 4137us) write_op.cc:312] APPLY: finished
0806 04:07:34.767628 (+ 85663us) log.cc:842] Serialized 35 byte log entry
0806 04:07:34.767824 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:34.767919 (+ 95us) write_op.cc:454] Released schema lock
0806 04:07:34.767981 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:34.782368 (+ 14387us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:34.785436 (+ 3068us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":206669,"reported_tablets":9,"spinlock_wait_cycles":1635376512,"child_traces":[["op",{"apply.queue_time_us":71738,"apply.run_cpu_time_us":3472,"apply.run_wall_time_us":116825,"num_ops":3,"prepare.queue_time_us":338723,"prepare.run_cpu_time_us":17337,"prepare.run_wall_time_us":190464,"raft.queue_time_us":100097,"raft.run_cpu_time_us":5044,"raft.run_wall_time_us":12901,"replication_time_us":408653,"thread_start_us":415618,"threads_started":4,"wal-append.queue_time_us":279220}]]}
I20260806 04:07:35.366868 4768 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.352327 7184 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.367715 7184 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.367928 4768 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.304592 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.347041 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.368759 7139 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.370119 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.352581 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.353336 4882 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.356663 4928 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.356776 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.343240 7199 consensus_queue.cc:1046] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:35.334985 4919 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:35.374778 7182 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.375389 7198 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.376076 7182 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.372385 4882 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.377665 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:35.377923 4882 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.372799 4876 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.371297 4768 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:35.379354 4923 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:35.380954 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.391824 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.370082 7190 raft_consensus.cc:1067] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:35.393973 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.373193 7219 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.377409 7003 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:35.376812 7080 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:35.404515 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.406509 7080 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.403576 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:35.407516 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:35.381091 7030 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.410169 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.407272 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.400929 4876 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.408939 5034 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:35.406555 7273 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.397858 7103 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.411912 4768 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.391224 7271 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.417625 4928 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:35.419132 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.420326 4882 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.405294 7274 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.422104 5859 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:45574
I20260806 04:07:35.422588 5859 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:35.422993 5859 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.409479 7080 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:35.411880 7001 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:35.426040 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.414335 7003 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:35.415180 7219 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:35.432598 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.418979 7103 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:35.438925 7103 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.439170 4882 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.433409 7003 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.440333 7219 raft_consensus.cc:686] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:35.440704 7003 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.428519 6851 raft_consensus.cc:3032] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:35.441833 6851 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:35.416208 7273 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:35.445999 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.447283 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.448472 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.449240 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.450371 6851 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.424005 7186 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.406849 7263 consensus_queue.cc:1046] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260806 04:07:35.482934 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.419906 7271 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:35.498234 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.499393 4808 raft_consensus.cc:2396] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:07:35.499334 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.500255 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.500973 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.478907 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.498467 7199 raft_consensus.cc:3032] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:35.442619 7190 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.479631 7274 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.506130 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.515282 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.448868 7001 raft_consensus.cc:686] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:35.517436 6851 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.498543 7271 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:35.509792 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 70730us
I20260806 04:07:35.449579 7273 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.510090 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.515280 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.515386 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.478434 7280 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:35.520162 4818 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
W20260806 04:07:35.452826 7137 consensus_queue.cc:1159] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7c76b6dcd8945cd818b8107caf4a174" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260806 04:07:35.529935 4882 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.531083 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.532460 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:35.526863 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 65050.5us
W20260806 04:07:35.528265 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 116283us
I20260806 04:07:35.510874 7199 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:35.540701 7001 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.542927 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.541882 7199 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.543931 7199 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.537817 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.537997 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.441359 7219 consensus_queue.cc:237] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.539482 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.548267 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.548899 7190 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.550169 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.554353 4996 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.556013 6891 consensus_queue.cc:1046] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:35.552860 4805 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:35.557124 7204 raft_consensus.cc:3032] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:35.557504 7204 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.558233 7278 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.558682 7278 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:35.563544 7278 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.561937 4818 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
W20260806 04:07:35.573936 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 53032.7us
I20260806 04:07:35.522095 7002 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.530318 7280 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.578089 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.581013 4882 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.587006 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.558393 7204 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.571251 6891 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.594409 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:35.594899 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.551432 7249 consensus_queue.cc:1046] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260806 04:07:35.596278 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.596693 4805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:35.597715 5816 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.599058 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.582919 7280 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.609159 4805 raft_consensus.cc:595] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:35.609997 4805 raft_consensus.cc:603] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:35.611965 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:07:35.614408 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:35.615769 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:35.621974 4930 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:35.637532 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.638372 5856 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.638885 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.639443 5028 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.640568 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.640879 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.681188 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.683048 5030 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:35.624634 7190 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.687271 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.687501 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.681046 7214 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:35.691277 5856 raft_consensus.cc:1226] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.608681 6891 consensus_queue.cc:1046] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260806 04:07:35.691975 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.701400 5812 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.703840 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.708206 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.696699 7190 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:07:35.720994 7285 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260806 04:07:35.721143 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:35.724117 7080 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.724910 7294 consensus_queue.cc:1046] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:35.726167 7294 raft_consensus.cc:3032] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:35.706501 7218 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:35.726529 7294 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.719455 7205 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:35.739277 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 158322us
I20260806 04:07:35.740597 5812 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.741662 4882 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.743927 5812 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:35.744962 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 172144us
I20260806 04:07:35.745664 4876 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.750025 7280 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.753047 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.753306 5028 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
W20260806 04:07:35.755113 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 157467us
I20260806 04:07:35.755445 5812 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.760535 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:35.762672 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:35.727471 7190 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.770041 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:35.778005 4923 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.778143 5038 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:35.755877 5816 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:35.774297 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 206690us
I20260806 04:07:35.699302 7292 consensus_queue.cc:1046] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260806 04:07:35.782052 4928 raft_consensus.cc:1226] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.781519 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.706641 7214 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.735021 7204 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.761706 7001 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.795392 7298 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.795397 7001 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.796654 7003 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.742461 7080 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.801156 4919 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:35.803385 7267 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.740164 5859 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.803714 7205 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:35.803426 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 47589.6us
I20260806 04:07:35.805172 7292 raft_consensus.cc:3032] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:35.805429 5859 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:35.805608 7292 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.805757 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.807394 5816 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.810428 7001 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.810925 7193 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:35.810886 7292 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.810887 7001 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.812191 7001 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.814131 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.814833 5816 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:35.815488 7205 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:35.815613 7284 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.817067 7205 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:35.817480 7205 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.817839 7205 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.816674 7003 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.819774 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:35.820801 6851 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:35.821137 6851 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:35.821552 6851 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.825778 7294 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.827733 7294 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:35.824491 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 56428.5us
I20260806 04:07:35.759562 7099 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:35.826506 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 70949us
I20260806 04:07:35.832983 5812 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.834211 4882 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:35.835093 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 50417.8us
I20260806 04:07:35.835529 4882 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.836771 6851 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.839416 4930 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.840358 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.763221 7002 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.841377 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:35.841964 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.842717 4805 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.808169 7184 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.844444 7184 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.844990 4924 raft_consensus.cc:1226] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.841302 7003 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.846138 7003 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:35.846587 7003 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.837687 4923 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:35.844812 4930 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.784489 7144 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:35.853199 4928 raft_consensus.cc:1226] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.851332 6851 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.812448 7292 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:35.854523 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:35.812983 7239 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 04:07:35.826607 7304 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.856549 4809 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.858075 4930 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.777395 7103 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.869063 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:35.856185 7144 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:35.808566 7182 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260806 04:07:35.865743 4920 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.879382 7214 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.884781 4810 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.890520 7184 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.891023 7184 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:35.891373 7184 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.896164 7103 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:35.901369 7277 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.420s user 0.001s sys 0.000s
I20260806 04:07:35.904028 7003 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.880244 7205 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.875079 7280 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.920321 7184 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.928994 7205 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:35.930106 7184 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:35.924424 7253 consensus_meta.cc:287] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.573s user 0.009s sys 0.009s
I20260806 04:07:35.910633 7249 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:35.931083 7249 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:35.931522 7249 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:35.880280 7002 raft_consensus.cc:2741] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260806 04:07:35.937038 7270 consensus_meta.cc:287] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.614s user 0.011s sys 0.000s
I20260806 04:07:35.937634 7270 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.941601 7249 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.943676 7249 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:35.908933 4805 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:35.956936 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 69742us
I20260806 04:07:35.876511 7144 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:35.972290 4814 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:35.967823 7202 raft_consensus.cc:3032] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:35.973287 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:35.965291 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 96272us
I20260806 04:07:35.974171 4926 raft_consensus.cc:1226] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:35.977092 7308 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:35.977324 4882 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.978056 7202 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.978040 7190 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.980877 7202 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:35.982362 7202 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:35.982949 4919 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:35.984773 7190 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:35.980152 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:35.964851 7257 consensus_meta.cc:287] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.605s user 0.005s sys 0.006s
I20260806 04:07:35.985270 7253 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:07:35.990003 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 153452us
I20260806 04:07:35.999565 4992 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:36.000555 6851 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.001108 4768 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.001786 6851 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.002242 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:35.998090 5032 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:36.003861 6851 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.004395 4992 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.005203 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:36.005827 6851 raft_consensus.cc:2741] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 04:07:36.006264 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.006968 6851 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.007855 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:36.008525 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.008916 4809 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:07:36.009387 4768 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.010440 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.011137 7214 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:36.011703 7214 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.011771 7293 raft_consensus.cc:3032] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:36.012367 7293 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:35.993687 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:36.013441 7293 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:35.997349 7253 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.015172 7253 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.014432 4674 catalog_manager.cc:5495] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260806 04:07:35.993978 7298 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:36.019111 7298 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:36.019588 7298 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:36.021196 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.022779 4768 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:36.024582 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 47349us
I20260806 04:07:36.025244 4882 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:36.026173 4882 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.027464 4882 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:35.971220 7273 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:36.040057 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 185609us
W20260806 04:07:36.044766 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 48862us
I20260806 04:07:35.976004 7199 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260806 04:07:36.057410 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.056661 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.056653 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.063630 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.043813 4804 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:35.998980 7218 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:36.056929 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.066413 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.074098 4808 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:36.057672 7003 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:36.045442 7030 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:36.075151 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.075387 7302 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.078346 4817 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:36.080031 4817 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:36.069712 5816 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.038830 7257 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.082604 4992 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.045907 7190 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:36.092130 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.085443 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.093338 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:35.987586 7278 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.045650 7219 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election won for term 2
W20260806 04:07:36.138682 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 57031.9us
I20260806 04:07:36.138821 4809 raft_consensus.cc:2379] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:36.087436 6851 consensus_queue.cc:1046] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:36.012370 7214 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:07:36.059036 7276 log.cc:924] Time spent T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.637s user 0.006s sys 0.000s
I20260806 04:07:36.045485 7103 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:36.045858 7280 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.090595 5816 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:36.136195 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 68614.3us
W20260806 04:07:36.160600 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 70126.3us
I20260806 04:07:36.160928 4996 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.162106 4996 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.162397 4882 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.136548 4931 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:36.137003 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 54513.9us
I20260806 04:07:36.163679 7193 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:36.164209 7193 raft_consensus.cc:686] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:36.165162 4992 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.045426 7182 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.165093 7193 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.065151 7184 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:36.166800 7293 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.166950 7184 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.141959 6851 raft_consensus.cc:3032] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:36.167964 6851 raft_consensus.cc:729] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.141652 7186 consensus_queue.cc:1046] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:36.142999 4668 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56998 (request call id 141) took 1999 ms. Trace:
W20260806 04:07:36.143407 7271 consensus_meta.cc:287] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.609s user 0.018s sys 0.000s
I20260806 04:07:36.087872 7218 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:07:36.158034 7274 consensus_meta.cc:287] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.640s user 0.011s sys 0.001s
I20260806 04:07:36.150416 4817 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:36.156960 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.158402 4925 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:36.152541 7139 consensus_meta.cc:287] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.783s user 0.005s sys 0.006s
I20260806 04:07:36.092303 7190 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:36.171361 4996 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:36.172940 7003 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:36.173030 7302 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:36.157258 6891 consensus_queue.cc:1046] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260806 04:07:36.173597 7288 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.173981 7257 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:36.173909 6851 consensus_queue.cc:260] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.174355 7312 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:36.176013 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 343083us
I20260806 04:07:36.174716 7002 consensus_queue.cc:1046] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:36.191550 5812 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.156360 7219 raft_consensus.cc:686] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.193823 7184 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:36.194057 6851 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.170518 7103 raft_consensus.cc:686] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.188663 7190 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:36.240464 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 49020.5us
I20260806 04:07:36.091635 7267 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.092s
I20260806 04:07:36.229288 7003 raft_consensus.cc:686] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:36.186951 7274 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.241504 5812 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.187712 7139 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.189416 7302 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.195806 7218 raft_consensus.cc:686] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:36.172610 4668 rpcz_store.cc:276] 0806 04:07:34.138372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:34.138715 (+ 343us) service_pool.cc:227] Handling call
0806 04:07:36.095054 (+1956339us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:35.412005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:35.423309 (+ 11304us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:35.423348 (+ 39us) write_op.cc:435] Acquired schema lock
0806 04:07:35.423364 (+ 16us) tablet.cc:660] Decoding operations
0806 04:07:35.428954 (+ 5590us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:35.429040 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:35.429065 (+ 25us) tablet.cc:683] Acquiring locks for 4 operations
0806 04:07:35.429488 (+ 423us) tablet.cc:699] Row locks acquired
0806 04:07:35.429513 (+ 25us) write_op.cc:260] PREPARE: finished
0806 04:07:35.429682 (+ 169us) write_op.cc:270] Start()
0806 04:07:35.429817 (+ 135us) write_op.cc:276] Timestamp: P: 1785989255429635 usec, L: 0
0806 04:07:35.429843 (+ 26us) op_driver.cc:331] REPLICATION: starting
0806 04:07:35.430492 (+ 649us) log.cc:842] Serialized 1798 byte log entry
0806 04:07:35.556343 (+125851us) op_driver.cc:444] REPLICATION: finished
0806 04:07:35.576964 (+ 20621us) write_op.cc:301] APPLY: starting
0806 04:07:35.577076 (+ 112us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:35.577280 (+ 204us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:35.577425 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:35.687171 (+109746us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:35.691478 (+ 4307us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0806 04:07:35.691534 (+ 56us) write_op.cc:312] APPLY: finished
0806 04:07:35.727579 (+ 36045us) log.cc:842] Serialized 41 byte log entry
0806 04:07:35.760568 (+ 32989us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:35.778919 (+ 18351us) write_op.cc:454] Released schema lock
0806 04:07:35.786099 (+ 7180us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:35.799089 (+ 12990us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:35.799136 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34104,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":20285,"apply.run_cpu_time_us":4089,"apply.run_wall_time_us":222547,"num_ops":4,"prepare.queue_time_us":96162,"prepare.run_cpu_time_us":3705,"prepare.run_wall_time_us":39857,"replication_time_us":126383,"thread_start_us":77757,"threads_started":2}]]}
I20260806 04:07:36.250691 4931 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:36.252642 4930 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:07:36.175779 7144 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.185915 7182 consensus_queue.cc:1046] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260806 04:07:36.204701 7214 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:36.276307 7280 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:36.275668 7326 consensus_queue.cc:1046] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260806 04:07:36.279767 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:36.280689 7003 consensus_queue.cc:237] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.281152 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:36.240409 7308 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:36.285386 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 105497us
W20260806 04:07:36.293708 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 91042.4us
I20260806 04:07:36.317644 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.204882 7270 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:36.183421 7271 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.300160 4925 raft_consensus.cc:3032] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 04:07:36.310111 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 134421us
I20260806 04:07:36.320995 4925 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:36.249493 7302 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:36.325423 5036 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:36.282691 7205 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.299423 7312 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:36.336892 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.336091 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.335105 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.325876 4919 raft_consensus.cc:2379] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:36.340337 5038 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:36.339735 5029 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:36.331172 4925 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.344720 6851 consensus_queue.cc:1046] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:07:36.327358 7001 consensus_meta.cc:287] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.515s user 0.013s sys 0.000s
I20260806 04:07:36.249107 7288 consensus_queue.cc:1046] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260806 04:07:36.351998 7283 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.351743 7218 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.352850 7273 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:36.352947 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:36.324072 7214 raft_consensus.cc:686] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.359776 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.328917 7186 raft_consensus.cc:3032] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:36.281984 7311 consensus_queue.cc:1046] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.177s
I20260806 04:07:36.353658 4804 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:36.376206 7204 consensus_meta.cc:287] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.576s user 0.007s sys 0.004s
I20260806 04:07:36.375768 7214 consensus_queue.cc:237] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.376725 7030 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.361218 4925 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:36.379364 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.378245 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.315567 7280 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:36.396557 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 58023.3us
I20260806 04:07:36.353474 7144 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:36.353969 7139 raft_consensus.cc:3032] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:36.347580 7312 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.407047 7318 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:36.367695 7186 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.373003 7278 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:36.320943 7219 consensus_queue.cc:237] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.349871 7263 consensus_queue.cc:1046] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260806 04:07:36.411443 4816 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:36.416534 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:36.417021 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:36.418258 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.418744 4931 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:36.420719 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.421022 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:36.421355 5032 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:36.421407 7318 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.421969 7318 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.422386 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:36.422947 5029 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.418505 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.420743 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:36.424968 4927 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:36.327929 7103 consensus_queue.cc:237] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:36.426690 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 255347us
I20260806 04:07:36.357718 7001 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.427099 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.430838 5036 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:36.411899 7030 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:36.427587 4996 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.430215 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.404922 7139 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.430883 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:36.458804 7315 consensus_queue.cc:1046] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260806 04:07:36.462563 7335 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260806 04:07:36.465699 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.439464 7030 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:36.467408 5037 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:36.475447 5028 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
W20260806 04:07:36.478880 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 51370.6us
I20260806 04:07:36.479753 4996 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.481894 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:36.465643 7204 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:36.475550 7030 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:36.482960 5034 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:36.507042 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.475342 5856 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:36.507233 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:36.481534 4919 raft_consensus.cc:1226] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:36.505374 7271 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:36.466296 7325 consensus_queue.cc:1046] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
I20260806 04:07:36.511705 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:36.513236 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.501152 7257 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.517477 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:36.521637 4927 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:36.478850 7186 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.523963 5028 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:36.529834 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:36.513520 7288 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.535578 4996 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.475075 7333 consensus_queue.cc:1046] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.164s
I20260806 04:07:36.527407 4879 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.506620 7204 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.466498 7297 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260806 04:07:36.544591 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.545328 4996 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.546170 7336 consensus_queue.cc:1046] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.129s
I20260806 04:07:36.511451 7139 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.537613 7280 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:36.555456 7204 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:36.557420 7342 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 04:07:36.557857 7137 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:36.559293 7199 raft_consensus.cc:3032] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:36.561609 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:36.543362 7288 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:36.548787 7271 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.563244 4996 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.557456 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.575099 4927 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:36.550349 7312 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:36.556723 7278 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.565758 7001 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:36.606837 7338 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.595532 7186 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.574321 7199 raft_consensus.cc:729] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.630766 5029 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:36.565316 7137 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.612453 4804 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:36.624310 7253 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.609s user 0.009s sys 0.004s
W20260806 04:07:36.614707 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 51514.5us
I20260806 04:07:36.636152 4996 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:36.623971 7294 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.790s user 0.007s sys 0.007s
I20260806 04:07:36.620348 4925 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:36.632761 7349 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:07:36.606886 7321 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260806 04:07:36.606000 7139 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:36.565590 7273 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 04:07:36.599267 7311 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.603768 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:36.635250 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.638559 7199 consensus_queue.cc:260] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.634243 7257 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.647655 7204 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.649187 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.618394 7312 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:36.601188 7302 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.655328 5030 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:36.609586 6891 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.681565 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.668452 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:36.684048 4768 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:36.672227 7316 raft_consensus.cc:3032] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:36.673220 7268 raft_consensus.cc:3032] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:36.674703 7184 raft_consensus.cc:979] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:36.640595 7253 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.671118 7099 raft_consensus.cc:3032] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:36.689723 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:36.685004 7199 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.678601 7315 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:36.676338 7202 raft_consensus.cc:3032] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:36.702183 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:36.707056 7355 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:36.695125 6891 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.745421 7326 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:36.745916 7326 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.746279 7326 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:36.708583 4996 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.708786 5854 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:36.693593 7325 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:36.690927 7268 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.718055 7337 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:36.686869 7204 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:36.736166 4813 raft_consensus.cc:1226] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:36.666388 7349 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.721202 7304 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:36.765285 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 56770.1us
I20260806 04:07:36.766386 4996 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.698870 7099 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:36.713888 7274 raft_consensus.cc:1067] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:36.685755 7312 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.695420 7316 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:36.772190 7302 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:36.772009 7268 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.725368 7202 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:36.736356 7001 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.778812 7304 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.753113 7325 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.693419 7190 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.747449 7315 raft_consensus.cc:686] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:36.750172 6891 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:36.792949 4996 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:36.754940 7355 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.691977 7137 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.773754 7204 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:36.807976 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 278666us
W20260806 04:07:36.809465 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 107371us
I20260806 04:07:36.773608 7337 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.815074 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:36.817011 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 59003.8us
I20260806 04:07:36.819691 4924 raft_consensus.cc:1226] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:36.821201 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:36.807087 7325 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.823607 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:36.824404 5029 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:36.825934 4930 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:36.829375 4768 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:36.832139 4809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:36.810279 7355 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.858938 5035 raft_consensus.cc:3032] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:36.859648 4925 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:36.861910 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.862881 7367 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:36.813110 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:36.864073 4920 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:36.863896 7315 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.864500 4926 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:36.831323 7267 consensus_queue.cc:1046] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260806 04:07:36.865028 7368 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.865617 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:36.871590 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1798) took 1276 ms. Trace:
I20260806 04:07:36.867949 5035 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:36.820255 7337 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:36.919878 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:36.946390 4808 raft_consensus.cc:2379] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:36.920133 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 52559us
I20260806 04:07:36.819509 7268 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:36.922976 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 56218.9us
I20260806 04:07:36.947244 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.811189 6891 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:36.951922 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:36.926779 7371 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.927160 7372 consensus_queue.cc:1046] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:36.929522 7378 consensus_queue.cc:1046] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:36.930850 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:36.933722 7253 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:36.933986 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:36.934154 7190 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:36.814563 7354 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260806 04:07:36.934894 7312 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:36.935940 7298 consensus_meta.cc:287] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.916s user 0.007s sys 0.007s
I20260806 04:07:36.936066 7369 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:36.831120 7293 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:36.940927 4818 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:36.940109 4816 rpcz_store.cc:276] 0806 04:07:35.584753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:35.584916 (+ 163us) service_pool.cc:227] Handling call
0806 04:07:36.860982 (+1276066us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54002a44d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:36.861118 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1490306304}
I20260806 04:07:36.955173 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:36.814764 7001 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:36.947871 7367 raft_consensus.cc:686] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:36.866977 7219 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260806 04:07:36.857605 7202 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.013456 5854 raft_consensus.cc:2379] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:37.011034 7298 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.016779 7369 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:36.867187 7099 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.012687 7372 consensus_queue.cc:1225] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:07:37.040136 7184 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.045047 7184 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.045915 7298 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.035635 7369 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:37.010190 7318 consensus_meta.cc:287] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.588s user 0.011s sys 0.005s
W20260806 04:07:37.031811 7352 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.271s user 0.001s sys 0.000s
I20260806 04:07:37.007503 7302 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260806 04:07:37.011593 7336 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260806 04:07:37.063831 7030 consensus_meta.cc:287] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.553s user 0.011s sys 0.001s
I20260806 04:07:37.076433 7349 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.115540 7001 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.116389 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1813) took 1058 ms. Trace:
W20260806 04:07:37.077319 7292 consensus_meta.cc:287] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.222s user 0.010s sys 0.005s
I20260806 04:07:37.066329 7184 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.111243 7030 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:37.122486 7307 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260806 04:07:36.931486 7316 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.038292 7367 consensus_queue.cc:237] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.195392 4813 rpcz_store.cc:276] 0806 04:07:36.058411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.076603 (+ 18192us) service_pool.cc:227] Handling call
0806 04:07:36.076950 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:36.755322 (+678372us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:37.116026 (+360704us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b54002a3ad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:37.116305 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1515492736}
W20260806 04:07:37.199087 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 331574us
I20260806 04:07:37.199021 7335 raft_consensus.cc:3032] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:37.230504 4927 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 04:07:37.233906 7379 consensus_queue.cc:1046] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260806 04:07:37.061427 7318 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.251695 7349 consensus_queue.cc:1046] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:37.259130 4920 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:37.263492 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:37.267563 4926 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:37.271585 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:37.236207 7335 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:37.265311 7278 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:07:37.291136 7199 consensus_meta.cc:287] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.569s user 0.014s sys 0.000s
I20260806 04:07:37.295476 7284 raft_consensus.cc:3032] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:37.296278 7284 raft_consensus.cc:729] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:37.296880 7326 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.546s user 0.008s sys 0.004s
I20260806 04:07:37.311893 7345 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:37.312505 7345 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:37.274581 6891 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:37.318073 7335 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.313484 7337 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.339671 4809 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:37.340236 4809 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:37.339907 7139 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:37.382838 7331 raft_consensus.cc:3032] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:37.383611 7331 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:37.297511 7326 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.401926 4804 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:37.402733 7337 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.403625 7326 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.404603 7288 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.284555 7278 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.405265 7288 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.409140 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:37.409672 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:37.418617 4768 leader_election.cc:334] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.566s (SENT)
I20260806 04:07:37.339670 7298 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.386828 7335 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:37.434258 7325 consensus_meta.cc:287] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.576s user 0.006s sys 0.011s
I20260806 04:07:37.382055 7355 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.447849 7355 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.400002 7268 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20260806 04:07:37.442590 7186 consensus_meta.cc:287] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.808s user 0.001s sys 0.019s
I20260806 04:07:37.452041 7030 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:37.298034 7284 consensus_queue.cc:260] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:37.456075 7360 log.cc:924] Time spent T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.658s user 0.007s sys 0.005s
I20260806 04:07:37.434065 4809 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.314157 7345 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.435904 7298 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.462623 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:37.454666 7331 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.466949 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 211) took 1216 ms. Trace:
I20260806 04:07:37.467581 7331 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:37.454334 7284 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:37.457126 7318 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:37.470248 7345 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:37.467576 4924 rpcz_store.cc:276] 0806 04:07:36.250730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.275968 (+ 25238us) service_pool.cc:227] Handling call
0806 04:07:36.312565 (+ 36597us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:36.820140 (+507575us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:37.466521 (+646381us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b5400413cd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:37.466777 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1165824256}
I20260806 04:07:37.481412 7298 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.436414 7325 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.457700 7288 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:37.324968 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:37.512470 4674 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46902 (request call id 282) took 3224 ms. Trace:
W20260806 04:07:37.528728 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.043s sys 0.093s
W20260806 04:07:37.529224 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.609s user 0.043s sys 0.093s
I20260806 04:07:37.538869 7288 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.539683 7288 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:37.553153 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.011s sys 0.013s
W20260806 04:07:37.553669 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.011s sys 0.013s
I20260806 04:07:37.521062 7316 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:37.561088 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.562785 5037 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:37.565496 5032 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:37.564671 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.566231 5038 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:37.566841 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.567446 5030 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:37.568157 5030 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:37.568812 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.569561 5026 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:37.573987 5029 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:37.564291 7288 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:37.512993 4674 rpcz_store.cc:276] 0806 04:07:34.287816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:34.288059 (+ 243us) service_pool.cc:227] Handling call
0806 04:07:34.369791 (+ 81732us) spinlock_profiling.cc:243] Waited 78.1 ms on lock 0x7b54001c1954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23df8b4d6c 00007f23df9aec3e 00007f23ddcc0fb4 00007f23ddcc0f42 00007f23ddcc0e71 00007f23ddcc0dec 00007f23ddcc00a2 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:37.512377 (+3142586us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:36.228600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:36.229291 (+ 691us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:36.229342 (+ 51us) write_op.cc:435] Acquired schema lock
0806 04:07:36.229365 (+ 23us) tablet.cc:660] Decoding operations
0806 04:07:36.229999 (+ 634us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:36.230109 (+ 110us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:36.230148 (+ 39us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:07:36.230390 (+ 242us) tablet.cc:699] Row locks acquired
0806 04:07:36.230416 (+ 26us) write_op.cc:260] PREPARE: finished
0806 04:07:36.230620 (+ 204us) write_op.cc:270] Start()
0806 04:07:36.230798 (+ 178us) write_op.cc:276] Timestamp: P: 1785989256230567 usec, L: 0
0806 04:07:36.230827 (+ 29us) op_driver.cc:331] REPLICATION: starting
0806 04:07:36.231507 (+ 680us) log.cc:842] Serialized 538 byte log entry
0806 04:07:36.460080 (+228573us) op_driver.cc:444] REPLICATION: finished
0806 04:07:36.666645 (+206565us) write_op.cc:301] APPLY: starting
0806 04:07:36.681192 (+ 14547us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:36.688777 (+ 7585us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:36.714114 (+ 25337us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:36.743868 (+ 29754us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:36.744063 (+ 195us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:07:36.744119 (+ 56us) write_op.cc:312] APPLY: finished
0806 04:07:36.744383 (+ 264us) log.cc:842] Serialized 23 byte log entry
0806 04:07:36.744565 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:36.744653 (+ 88us) write_op.cc:454] Released schema lock
0806 04:07:36.744695 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:36.744967 (+ 272us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:36.744997 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":43218,"reported_tablets":8,"spinlock_wait_cycles":179687168,"child_traces":[["op",{"apply.queue_time_us":167378,"apply.run_cpu_time_us":2358,"apply.run_wall_time_us":80228,"num_ops":1,"prepare.queue_time_us":210911,"prepare.run_cpu_time_us":3144,"prepare.run_wall_time_us":9530,"replication_time_us":200252,"spinlock_wait_cycles":1559936,"thread_start_us":357972,"threads_started":2}]]}
I20260806 04:07:37.582574 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.591492 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.594898 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.602103 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.605050 5029 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:37.596462 5813 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.012s sys 0.016s
W20260806 04:07:37.608134 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.015s sys 0.031s
W20260806 04:07:37.608601 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.015s sys 0.031s
W20260806 04:07:37.603418 4769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.019s sys 0.039s
W20260806 04:07:37.609058 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 749671us
I20260806 04:07:37.616499 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.618595 5037 raft_consensus.cc:1226] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:37.619060 5028 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:37.619946 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.620970 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.621456 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.622166 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.623150 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.624177 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.624399 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.624997 5037 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:37.625456 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.626118 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.626773 5037 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:37.629993 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.631150 5030 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
W20260806 04:07:37.609603 5813 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.013s sys 0.017s
W20260806 04:07:37.632886 4769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.019s sys 0.039s
I20260806 04:07:37.603703 7278 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:37.639410 7193 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.602975 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:37.650707 4771 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.012s sys 0.094s
W20260806 04:07:37.659914 4771 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.752s user 0.012s sys 0.095s
I20260806 04:07:37.668412 7356 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:37.669936 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:37.673588 7369 consensus_meta.cc:287] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.626s user 0.005s sys 0.012s
I20260806 04:07:37.674095 7369 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.676811 7369 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.669380 7356 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.680603 7356 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.683727 7205 raft_consensus.cc:3032] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:37.684315 7205 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:37.689890 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:37.693077 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.693503 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.698930 5026 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:37.698988 4816 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:37.700750 4813 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:37.699818 7139 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:37.701453 7139 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.702154 4813 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:37.694722 4814 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 04:07:37.704870 4992 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.023s sys 0.035s
W20260806 04:07:37.705194 4992 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.782s user 0.023s sys 0.035s
I20260806 04:07:37.706359 5038 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:37.707531 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:37.700650 4996 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:37.668563 7325 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:37.657608 5032 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:37.710078 4996 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:37.710574 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.688184 7278 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.697073 5029 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:37.711925 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.729856 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.700250 7214 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:37.685420 7205 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.722273 4816 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:37.700430 7288 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:37.732710 5867 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:37.734951 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:37.744463 4882 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.033s sys 0.045s
W20260806 04:07:37.744920 4882 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.674s user 0.033s sys 0.045s
I20260806 04:07:37.745046 5029 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:37.745743 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 674803us
I20260806 04:07:37.745894 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:37.735597 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.710955 7268 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.736799 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.738952 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:37.698274 7030 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 04:07:37.800097 5032 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:37.753358 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 307960us
I20260806 04:07:37.752410 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:37.691080 7193 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:37.805006 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 52690.4us
I20260806 04:07:37.727283 7318 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:37.716492 7382 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:37.731585 7304 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:37.817472 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:37.817745 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.818988 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.820731 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.733721 7316 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:37.824254 5030 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:37.828482 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.829428 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:37.805037 7205 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:37.831319 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:37.834630 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:37.835137 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.836644 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:37.837648 5032 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260806 04:07:37.839123 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 186345us
I20260806 04:07:37.840601 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:37.820629 7315 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.820713 7204 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:07:37.848688 4996 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:37.735067 7278 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.885768 7202 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:37.831601 7337 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:37.831045 7325 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.832399 7388 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:37.864867 7139 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.871865 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:37.872404 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:37.869751 7219 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.867130 7198 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:37.839279 6970 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:37.838697 5859 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:37.868821 7326 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260806 04:07:37.877111 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:37.891412 7257 raft_consensus.cc:2741] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 04:07:37.878643 4920 raft_consensus.cc:1226] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:37.851812 4814 raft_consensus.cc:1226] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:37.879848 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:37.881177 7400 log.cc:924] Time spent T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260806 04:07:37.882128 7273 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.883761 7137 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:37.885608 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 192) took 1056 ms. Trace:
I20260806 04:07:37.868364 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:37.886765 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 46160.3us
I20260806 04:07:37.803896 7268 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.839586 7345 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:37.890540 4876 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:37.895303 7267 raft_consensus.cc:3032] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:37.897617 7315 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.901600 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 192) took 1557 ms. Trace:
I20260806 04:07:37.901695 7099 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:37.903635 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1445) took 2263 ms (client timeout 2012 ms). Trace:
I20260806 04:07:37.904652 5035 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.736367 7214 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.908376 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1837) took 1157 ms. Trace:
I20260806 04:07:37.890213 4996 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:37.917999 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 68688.5us
I20260806 04:07:37.838292 7316 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.980213 7214 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.980844 7316 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:37.867802 7218 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 04:07:37.907979 7381 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.982303 7381 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:37.957748 4805 rpcz_store.cc:276] 0806 04:07:36.344076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.344206 (+ 130us) service_pool.cc:227] Handling call
0806 04:07:36.375544 (+ 31338us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:36.375941 (+ 397us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:07:36.375969 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:36.376412 (+ 443us) log.cc:842] Serialized 26 byte log entry
0806 04:07:37.290304 (+913892us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b180035a880. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e1411
0806 04:07:37.290478 (+ 174us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:37.290516 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:37.290584 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:37.290650 (+ 66us) raft_consensus.cc:1671] finished
0806 04:07:37.290681 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:37.901330 (+610649us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b5400271ad0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:37.901494 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7678,"raft.queue_time_us":3070,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":10639,"spinlock_wait_cycles":41789056,"thread_start_us":24119,"threads_started":1,"wal-append.queue_time_us":83389,"wal-append.run_cpu_time_us":2290,"wal-append.run_wall_time_us":1117539}
I20260806 04:07:37.984375 7381 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.984261 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.984721 7278 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:37.985308 7399 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260806 04:07:37.985378 7278 consensus_queue.cc:1046] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:37.987231 7404 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:37.988189 7393 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:37.988760 7393 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.936515 4930 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:37.993983 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 103864us
I20260806 04:07:37.937659 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.995034 4814 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:37.996402 4814 raft_consensus.cc:1226] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:37.938637 4809 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.907274 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:37.946040 5036 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:37.919260 7219 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.947755 4813 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:37.950380 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:37.955200 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:37.955482 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 65032.2us
I20260806 04:07:38.012868 4876 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.014055 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:37.958297 4816 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 3. Ops: []
I20260806 04:07:37.971915 5035 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:37.963402 7407 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:37.978596 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:37.963546 5034 rpcz_store.cc:269] 0806 04:07:35.640497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:35.640644 (+ 147us) service_pool.cc:227] Handling call
0806 04:07:37.903386 (+2262742us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540032f0d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:37.903521 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1694547328}
I20260806 04:07:37.972539 4804 rpcz_store.cc:276] 0806 04:07:36.751135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.751308 (+ 173us) service_pool.cc:227] Handling call
0806 04:07:36.771986 (+ 20678us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:37.402423 (+630437us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:37.908081 (+505658us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b54003c1c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:37.908303 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1448091392}
I20260806 04:07:37.911705 7325 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.002907 7316 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.004418 7408 raft_consensus.cc:3032] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:37.956741 7267 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:37.972127 7409 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.005700 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:38.001793 7381 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.008548 7193 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:37.957163 7253 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.009083 7393 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:37.971339 7308 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260806 04:07:38.010948 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:33548 (request call id 1838) took 1181 ms. Trace:
I20260806 04:07:38.020953 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:38.021173 4996 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.024272 7218 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.036656 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1836) took 1290 ms. Trace:
I20260806 04:07:37.951613 7273 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.045428 4814 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:38.015576 7214 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:38.094959 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 73891us
I20260806 04:07:38.121505 4996 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.122479 7257 raft_consensus.cc:2741] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 04:07:38.123710 7257 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:38.022166 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 67040.1us
I20260806 04:07:38.124171 7257 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.034044 7408 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:38.125790 7408 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.126760 7257 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.126914 7408 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:38.127774 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:38.044247 7335 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.606s user 0.012s sys 0.001s
I20260806 04:07:38.132270 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:38.132534 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.133576 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.133863 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.134546 5032 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
W20260806 04:07:38.111263 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 242666us
I20260806 04:07:37.971465 7001 consensus_queue.cc:1046] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260806 04:07:38.048290 4809 rpcz_store.cc:276] 0806 04:07:36.816570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.816724 (+ 154us) service_pool.cc:227] Handling call
0806 04:07:37.894395 (+1077671us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b540039bcd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9951da7 00007f23de89dee0 00007f23d7f8f784 00007f23d7f8f712 00007f23d7f8f641 00007f23d7f8f5bc 00007f23d7f8e872 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:37.998166 (+103771us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105975040}
I20260806 04:07:38.136214 7335 raft_consensus.cc:3032] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:38.136703 7335 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:38.035959 7382 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:38.116290 4810 rpcz_store.cc:276] 0806 04:07:36.746452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.748970 (+ 2518us) service_pool.cc:227] Handling call
0806 04:07:36.749405 (+ 435us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:36.751991 (+ 2586us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:36.752031 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:36.752085 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:36.752125 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:38.036330 (+1284205us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b54003d8954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:38.036395 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:38.036578 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5394688}
I20260806 04:07:38.137738 5812 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:38.138586 7001 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:38.139159 7001 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:38.137861 7382 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.139681 7001 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:38.141876 7382 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.045207 7409 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:38.126263 5034 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:38.115823 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 155547us
I20260806 04:07:38.148082 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.148993 4810 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:38.150457 4996 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.154944 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.155731 5034 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:38.114498 5857 raft_consensus.cc:2379] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:38.047194 7410 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:38.159785 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:38.157258 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:38.161319 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.162086 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:38.164352 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:38.164917 5034 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.160465 5863 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:38.166688 5032 raft_consensus.cc:2362] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2. Re-sending same reply.
I20260806 04:07:38.118171 7316 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:38.037168 7204 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260806 04:07:38.044898 7381 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.171123 4995 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:38.181504 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.182191 5030 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:38.137811 7335 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:38.167136 7284 consensus_meta.cc:287] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.692s user 0.012s sys 0.000s
I20260806 04:07:38.186336 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.035110 7407 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.188745 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.180333 7316 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:38.170796 7139 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.192457 7381 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.157061 7214 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.195801 7219 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.196520 7418 consensus_queue.cc:1046] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:38.170620 7198 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.094079 7253 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.220826 7273 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.222515 7253 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.161204 7382 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:38.202965 4995 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.185415 7193 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:38.223536 7193 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:38.223971 7193 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:38.224068 7257 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.189669 7336 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.158385 7315 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.187860 7288 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:38.197525 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 149109us
I20260806 04:07:38.158908 7267 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.219987 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.223345 7382 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.205436 5034 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:38.206282 7388 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:38.235416 5028 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:38.234457 7336 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.224785 4669 catalog_manager.cc:5495] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 2, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 2 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:38.238101 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.239221 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.240243 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.208169 7316 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:38.241222 4669 catalog_manager.cc:5495] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 3 to 4, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 4 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:38.242642 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.208644 7409 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.246124 5030 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:38.249548 4876 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.216898 7268 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.256982 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:38.239187 7388 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:38.236635 7382 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:38.237033 7335 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.236364 7288 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:38.209496 7139 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:37.947396 5856 rpcz_store.cc:276] 0806 04:07:36.829117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.830151 (+ 1034us) service_pool.cc:227] Handling call
0806 04:07:36.830522 (+ 371us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:37.220526 (+390004us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:37.222994 (+ 2468us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:37.223958 (+ 964us) log.cc:842] Serialized 26 byte log entry
0806 04:07:37.610739 (+386781us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:37.629986 (+ 19247us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:37.635220 (+ 5234us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:37.885304 (+250084us) raft_consensus.cc:1671] finished
0806 04:07:37.885333 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:37.885506 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27231,"raft.queue_time_us":91559,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":17723,"thread_start_us":636689,"threads_started":2,"wal-append.queue_time_us":597038}
I20260806 04:07:38.269310 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:38.261914 7349 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:38.258558 7396 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.232673 7198 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.265669 7388 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.265795 7315 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.266868 7325 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.284138 4768 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:38.270376 7268 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.268606 7288 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.293078 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.274474 7418 consensus_queue.cc:1046] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260806 04:07:38.295485 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:38.295886 7416 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.249923 7393 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260806 04:07:38.292840 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.264415 7284 consensus_queue.cc:1046] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260806 04:07:38.303484 4931 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:38.314905 4930 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:38.304988 7419 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:38.322243 7429 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.323714 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.324481 4925 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:38.325562 7338 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.326025 7139 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.326553 7198 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.291178 7349 raft_consensus.cc:686] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:38.250717 7409 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.272691 7337 consensus_queue.cc:1046] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260806 04:07:38.361477 5035 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:38.317058 4919 raft_consensus.cc:3032] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:38.339303 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.313915 7219 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.341194 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.379134 4931 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:38.342334 4923 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:38.337266 7349 consensus_queue.cc:237] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.354370 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:38.355705 4926 raft_consensus.cc:1261] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:38.359246 7371 raft_consensus.cc:3032] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:38.328913 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.364334 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1472) took 1539 ms. Trace:
I20260806 04:07:38.327847 7257 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.368988 4813 raft_consensus.cc:2396] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 04:07:38.365883 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.430064 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0940457704114112b22dab24a18b43b6"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:38.437490 4923 raft_consensus.cc:595] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.433053 5035 rpcz_store.cc:276] 0806 04:07:36.824510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.824706 (+ 196us) service_pool.cc:227] Handling call
0806 04:07:36.825004 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:37.904186 (+1079182us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400204a20. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d98f0951 00007f23d9951321 00007f23d9952b4e 00007f23d995cc89 00007f23d995d556 00007f23d9959edf 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612
0806 04:07:38.363934 (+459748us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:38.364262 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251554688}
I20260806 04:07:38.303411 7315 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.433091 4930 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:38.435917 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.263067 7423 consensus_queue.cc:1046] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260806 04:07:38.326481 7338 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:38.371369 7182 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260806 04:07:38.451432 4928 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:38.452838 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.364759 7429 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:38.366947 4919 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:38.419092 7103 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.443800 7257 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.468077 4920 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 04:07:38.431495 7371 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:38.322189 7419 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.439640 7199 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.464506 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:38.453454 7418 raft_consensus.cc:3032] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:38.352375 7426 consensus_queue.cc:1046] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.205s
I20260806 04:07:38.460309 7304 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.497567 5034 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:38.289064 7396 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.522675 7311 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.454589 7393 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260806 04:07:38.479012 7214 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.533746 7431 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:07:38.497653 7429 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:38.538239 7422 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260806 04:07:38.547870 7336 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.497637 7199 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.553907 7338 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.497925 7379 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.526139 7184 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:38.493530 7418 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:38.540664 7331 consensus_meta.cc:287] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.068s user 0.011s sys 0.005s
I20260806 04:07:38.562453 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.487083 7103 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:38.571005 7137 consensus_meta.cc:287] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.618s user 0.015s sys 0.000s
I20260806 04:07:38.521688 7419 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.592001 7419 raft_consensus.cc:3032] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:38.592789 7419 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:38.496836 4919 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.603904 5038 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 04:07:38.604362 5038 pending_rounds.cc:76] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Aborting all ops after (but not including) 1
I20260806 04:07:38.525514 7311 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.604611 5038 pending_rounds.cc:97] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 04:07:38.564880 7431 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:38.494958 7407 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.632368 5038 raft_consensus.cc:2866] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:38.559952 7379 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.509675 7304 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.567286 7184 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.631181 7418 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.636689 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:38.629000 7284 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.602533 7378 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.595860 7336 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260806 04:07:38.649035 7409 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.651537 7390 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:38.652438 7103 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.653506 4919 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.603088 7371 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.652525 7315 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.677620 4995 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.677692 7184 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.677591 7379 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.678277 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.678719 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.686096 4813 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:38.681418 7199 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.688680 4996 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.689687 7204 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:38.690120 7204 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.690549 7204 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:38.691536 7429 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.686218 4818 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
W20260806 04:07:38.669417 7099 consensus_meta.cc:287] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.698s user 0.009s sys 0.006s
I20260806 04:07:38.695823 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.670178 7338 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.681789 7331 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:38.651885 7431 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.706897 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.663151 7284 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.707710 5867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 186) took 2233 ms. Trace:
I20260806 04:07:38.695812 7418 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.711018 5867 rpcz_store.cc:276] 0806 04:07:36.474600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.474774 (+ 174us) service_pool.cc:227] Handling call
0806 04:07:36.475106 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:37.744518 (+1269412us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:38.707490 (+962972us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540059ecd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:38.707657 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2891744512}
I20260806 04:07:38.705184 7205 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.717979 7205 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.699507 7371 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:38.700800 7379 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:38.661355 7378 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.732544 7383 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:38.733309 7383 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.705865 7396 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.703935 7338 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.730165 7379 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:38.753024 7205 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.751570 4808 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:38.740772 7431 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:38.764622 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.002s sys 0.001s
W20260806 04:07:38.765069 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.002s sys 0.001s
W20260806 04:07:38.765364 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 515329us
I20260806 04:07:38.769217 4817 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:38.769426 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.770393 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.694046 7419 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.770979 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:38.771548 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.777253 4813 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3. Re-sending same reply.
I20260806 04:07:38.771723 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:38.771178 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 487090us
I20260806 04:07:38.778316 4804 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:38.778579 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.779428 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.781780 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.777498 4817 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:38.782461 4923 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:38.784492 4882 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:38.780211 5035 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:38.755301 7304 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.756964 7423 raft_consensus.cc:3032] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:38.752712 7379 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.788175 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:38.786404 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 497241us
I20260806 04:07:38.794538 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.747058 7311 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.771601 7409 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:38.799717 5028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:38.801273 5028 raft_consensus.cc:595] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:38.801693 5028 raft_consensus.cc:603] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 04:07:38.795120 7408 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.663s user 0.011s sys 0.004s
W20260806 04:07:38.806404 7001 consensus_meta.cc:287] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.666s user 0.016s sys 0.000s
I20260806 04:07:38.790541 7383 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.807396 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.807634 7288 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.808883 7338 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:38.788123 7423 raft_consensus.cc:729] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:38.818521 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:38.812896 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.821180 7184 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:07:38.821825 7193 consensus_meta.cc:287] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.597s user 0.009s sys 0.005s
I20260806 04:07:38.823079 7218 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 04:07:38.824316 7316 consensus_meta.cc:287] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.566s user 0.011s sys 0.001s
I20260806 04:07:38.842209 4924 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:38.852311 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.846980 5028 raft_consensus.cc:1261] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:38.787738 7429 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 04:07:38.856326 4920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:38.857103 4920 raft_consensus.cc:595] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:38.858603 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:38.871157 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 861864us
I20260806 04:07:38.851174 7103 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:38.849512 7218 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:07:38.862973 7382 consensus_meta.cc:287] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.590s user 0.011s sys 0.004s
I20260806 04:07:38.827291 7001 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:38.871191 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 379873us
I20260806 04:07:38.896436 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.853765 7184 raft_consensus.cc:686] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:38.909657 7184 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.911701 7288 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:38.911108 7423 consensus_queue.cc:260] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:38.872157 7284 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:38.912730 7423 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:38.913012 7219 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:38.913676 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.914268 4931 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:38.899667 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:38.891687 4810 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.023s sys 0.032s
I20260806 04:07:38.900645 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.901254 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.902216 4804 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:38.873991 7103 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:38.905066 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:38.865578 7337 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260806 04:07:38.242262 7308 consensus_queue.cc:1159] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "99690415a7b3441fbdb5e943667556a4" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:07:38.869136 7214 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:38.855857 7316 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.932325 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:38.850555 7193 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.924365 4920 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:38.925221 4930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:38.945829 4930 raft_consensus.cc:595] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:38.894114 7416 consensus_meta.cc:287] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.567s user 0.004s sys 0.012s
I20260806 04:07:38.923919 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.897154 5028 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
W20260806 04:07:38.916610 4810 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.023s sys 0.032s
I20260806 04:07:38.927774 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:38.948482 5037 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:38.949561 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:07:38.915076 7390 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:38.934029 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:38.914602 7382 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:38.936370 7335 consensus_meta.cc:287] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.668s user 0.009s sys 0.005s
I20260806 04:07:38.929502 7214 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:38.925631 7103 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:38.951779 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:38.973577 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:38.978150 5812 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:38.930051 4669 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.005s sys 0.027s
I20260806 04:07:38.925613 7404 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:38.957993 4809 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:38.967787 5029 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:38.947796 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1477) took 1209 ms. Trace:
I20260806 04:07:38.956523 7416 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:38.989681 5026 rpcz_store.cc:276] 0806 04:07:37.738636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.804804 (+ 66168us) service_pool.cc:227] Handling call
0806 04:07:37.805305 (+ 501us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:37.991859 (+186554us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:37.991902 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:37.991957 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:37.991997 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:38.947545 (+955548us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b540037df54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:38.947594 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:38.947736 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":428338432}
I20260806 04:07:39.003532 4816 raft_consensus.cc:1226] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:38.971218 7367 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.008s sys 0.044s
I20260806 04:07:38.973448 7214 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:38.989912 4669 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.707s user 0.005s sys 0.027s
I20260806 04:07:38.947549 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:38.968844 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:38.988853 5032 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.005s sys 0.018s
I20260806 04:07:38.967159 4919 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:39.020678 5812 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:39.021272 5032 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.005s sys 0.018s
I20260806 04:07:39.009737 7335 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:39.092720 7311 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:39.104147 7388 consensus_meta.cc:287] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.800s user 0.013s sys 0.004s
I20260806 04:07:39.022727 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:39.018968 7367 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s user 0.008s sys 0.045s
I20260806 04:07:39.107060 7388 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.100963 7335 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.002802 7404 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:38.967907 7390 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.128168 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1450) took 1285 ms. Trace:
I20260806 04:07:39.112507 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.061873 7447 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:39.110911 7304 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.110940 7288 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.102025 7311 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:39.131489 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 106731us
I20260806 04:07:39.133713 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1485) took 1321 ms. Trace:
I20260806 04:07:39.128679 4927 rpcz_store.cc:276] 0806 04:07:37.859276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.870441 (+ 11165us) service_pool.cc:227] Handling call
0806 04:07:38.344342 (+473901us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fab7c 00007f23d69fa73b 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:38.344627 (+ 285us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:38.357847 (+ 13220us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:38.357880 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:38.364370 (+ 6490us) log.cc:842] Serialized 26 byte log entry
0806 04:07:38.870275 (+505905us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:38.870330 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:38.870404 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:39.127777 (+257373us) raft_consensus.cc:1671] finished
0806 04:07:39.127796 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:39.128101 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38980,"raft.queue_time_us":87133,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":11475,"spinlock_wait_cycles":6431360,"thread_start_us":419802,"threads_started":1,"wal-append.queue_time_us":612008}
I20260806 04:07:39.152493 5812 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:39.154224 5862 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:39.130383 7335 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:39.152174 5038 rpcz_store.cc:276] 0806 04:07:37.812231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.840588 (+ 28357us) service_pool.cc:227] Handling call
0806 04:07:37.867803 (+ 27215us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:38.633031 (+765228us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.133423 (+500392us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b54002d1ed4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:39.133637 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1621965696}
I20260806 04:07:39.135480 4805 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:39.139231 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.135658 7447 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:39.144069 7331 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.144547 7193 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.144235 7378 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.135430 7396 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.129470 7390 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.145665 7001 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:39.175752 4816 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:39.151460 7338 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.177640 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:39.141729 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:39.181243 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 386643us
I20260806 04:07:39.181888 5816 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:39.185741 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:39.016981 4876 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.000s sys 0.041s
W20260806 04:07:39.196375 4876 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.946s user 0.000s sys 0.041s
I20260806 04:07:39.150296 7304 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:39.179343 7426 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.028s sys 0.040s
I20260806 04:07:39.155985 7382 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.156642 7312 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.194553 4770 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:39.198244 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:39.198287 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 948763us
W20260806 04:07:39.191258 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 315302us
I20260806 04:07:39.186249 7356 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:39.184485 7447 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:39.154230 7288 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.231016 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:39.233476 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:39.233907 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:39.243386 4768 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:39.236495 4816 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
W20260806 04:07:39.227007 7426 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.584s user 0.028s sys 0.041s
W20260806 04:07:39.221702 7393 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.039s sys 0.099s
I20260806 04:07:39.186394 7331 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.232825 7298 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.228370 4770 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:39.217867 7316 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.227314 7186 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:39.237671 7283 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:39.320390 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 136904us
W20260806 04:07:39.323922 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 95637.6us
I20260806 04:07:39.233410 7356 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:39.252847 7404 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.217231 7193 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260806 04:07:39.192696 7396 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.242110 7378 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:39.255781 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:39.306032 4992 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.571s (SENT) [suppressed 52 similar messages]
I20260806 04:07:39.235463 7001 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:39.255676 7337 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:39.315398 7182 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.242365 7382 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:39.255847 7349 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:39.265756 4768 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:39.295087 7204 consensus_meta.cc:287] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.604s user 0.015s sys 0.005s
I20260806 04:07:39.251211 5859 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:39.197546 7338 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.236924 7312 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.302094 7253 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:39.317245 7354 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.020s
I20260806 04:07:39.288932 7450 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:07:39.290570 7393 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.040s sys 0.099s
W20260806 04:07:39.374939 7454 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.082s user 0.002s sys 0.000s
I20260806 04:07:39.332027 7356 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.376991 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 222) took 1040 ms. Trace:
I20260806 04:07:39.377555 7396 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.343889 7337 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:39.326031 7461 consensus_queue.cc:1046] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260806 04:07:39.380173 7337 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.377421 4926 rpcz_store.cc:276] 0806 04:07:38.336213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.445336 (+109123us) service_pool.cc:227] Handling call
0806 04:07:38.484335 (+ 38999us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:38.495450 (+ 11115us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:38.495481 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:38.507808 (+ 12327us) log.cc:842] Serialized 26 byte log entry
0806 04:07:38.949671 (+441863us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:38.949717 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:38.949781 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:39.376737 (+426956us) raft_consensus.cc:1671] finished
0806 04:07:39.376758 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:39.376929 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10681,"raft.queue_time_us":106704,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":199,"thread_start_us":383247,"threads_started":1,"wal-append.queue_time_us":673854}
I20260806 04:07:39.381390 7461 raft_consensus.cc:3032] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:39.361831 7268 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.346369 7311 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.329638 7283 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:07:39.350317 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 94627.4us
I20260806 04:07:39.331169 7371 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:07:39.352586 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 644046us
I20260806 04:07:39.332623 7186 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:39.356989 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 91348.7us
I20260806 04:07:39.398576 4876 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.398928 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:39.360209 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.358521 4879 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.291965 7416 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260806 04:07:39.361511 5866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 520) took 1384 ms. Trace:
I20260806 04:07:39.340684 7388 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.357385 7349 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:39.364042 7382 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.355312 7030 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:39.345738 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:39.323158 7298 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.382010 7461 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:39.343423 7001 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:07:39.354631 4923 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.018s sys 0.028s
W20260806 04:07:39.480888 4923 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.683s user 0.019s sys 0.028s
I20260806 04:07:39.399966 4768 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:39.482012 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 82132us
I20260806 04:07:39.482975 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:39.404096 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 45665.2us
I20260806 04:07:39.361615 7182 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.432662 4876 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.403861 5866 rpcz_store.cc:276] 0806 04:07:37.977212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.977440 (+ 228us) service_pool.cc:227] Handling call
0806 04:07:38.430466 (+453026us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:38.430808 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:38.485662 (+ 54854us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:38.485702 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:38.485757 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:38.485797 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.361220 (+875423us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b54004448d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:39.361296 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:39.361443 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158276224}
I20260806 04:07:39.401492 7311 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.386533 7455 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.383572 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:39.462663 4927 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.393043 7283 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:39.390704 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.352227 7378 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.361544 7204 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.398368 7326 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.401065 7396 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:39.441322 7379 consensus_meta.cc:287] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.642s user 0.010s sys 0.003s
W20260806 04:07:39.489018 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 56520.4us
W20260806 04:07:39.447340 7458 log.cc:924] Time spent T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.097s user 0.001s sys 0.001s
I20260806 04:07:39.438640 7331 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.440649 7312 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.501575 7415 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:39.503087 7463 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:39.504958 7103 consensus_meta.cc:287] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.506s user 0.012s sys 0.001s
I20260806 04:07:39.444999 7450 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:39.504209 7311 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.405884 7338 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.514539 4930 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:39.442495 7468 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:39.460929 7349 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:39.444041 7316 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.458321 7371 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:39.565845 7371 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:39.555750 7103 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.566481 7371 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.569301 7103 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.570943 7298 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:39.571513 7469 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20260806 04:07:39.572099 7450 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.458101 7318 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.389782 7268 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.533478 4876 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.487622 7001 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.576606 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:39.578085 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:39.478973 7356 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.516803 7304 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.539139 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.541399 7331 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.542757 7312 consensus_queue.cc:1046] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:39.548502 7182 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.532018 7416 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:39.563264 5036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1747) took 1935 ms (client timeout 1541 ms). Trace:
I20260806 04:07:39.597541 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:39.543267 7186 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.564441 7316 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.575120 4804 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:39.579887 7278 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:39.585124 5857 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
W20260806 04:07:39.597849 5036 rpcz_store.cc:269] 0806 04:07:37.615938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.616070 (+ 132us) service_pool.cc:227] Handling call
0806 04:07:39.551464 (+1935394us) spinlock_profiling.cc:243] Waited 94 ms on lock 0x7b5400016654. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:39.551597 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216258304}
I20260806 04:07:39.580080 7318 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.579160 7288 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.544093 7396 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.578293 7450 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.585835 7351 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:39.539942 7379 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.597870 7193 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260806 04:07:39.555243 7326 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.601063 7278 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.597378 7204 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.580739 7463 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:39.578393 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.525862 7455 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.594317 7311 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.640967 5035 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:39.577210 7415 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.571830 7461 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.619515 7467 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:39.621466 7316 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.621907 7257 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.604393 7003 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:39.599187 4876 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.648344 7326 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.680356 7003 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:39.648880 7356 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.649727 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.656323 7268 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.657158 7415 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.660220 7378 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.661809 7315 raft_consensus.cc:3032] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:39.700704 7315 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:39.702752 7315 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.661881 7382 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.663487 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 208) took 1688 ms. Trace:
I20260806 04:07:39.627458 7278 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.596098 7331 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.665952 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:39.708854 4924 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.666580 4927 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:39.664045 7379 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.683324 7003 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:39.708050 7455 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.708370 7001 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.684928 7467 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:39.686496 5853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 198) took 1458 ms. Trace:
W20260806 04:07:39.684159 7423 consensus_meta.cc:287] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.771s user 0.010s sys 0.008s
I20260806 04:07:39.712540 7467 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:39.712566 5853 rpcz_store.cc:276] 0806 04:07:38.227624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.539331 (+1311707us) service_pool.cc:227] Handling call
0806 04:07:39.686345 (+147014us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:39.705071 4814 rpcz_store.cc:276] 0806 04:07:37.975500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.999554 (+ 24054us) service_pool.cc:227] Handling call
0806 04:07:37.999881 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:38.118257 (+118376us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.663193 (+1544936us) spinlock_profiling.cc:243] Waited 45.2 ms on lock 0x7b540039a8d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:39.663418 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":103949568}
I20260806 04:07:39.714721 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:39.719672 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.725185 5859 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:39.714572 7418 raft_consensus.cc:2741] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 04:07:39.732852 7379 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:39.733569 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 134474us
I20260806 04:07:39.734334 7331 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.735206 4876 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:39.696952 7450 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:39.789479 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1150) took 2891 ms (client timeout 2205 ms). Trace:
W20260806 04:07:39.750598 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1497) took 1520 ms (client timeout 1935 ms). Trace:
W20260806 04:07:39.809191 5034 rpcz_store.cc:269] 0806 04:07:38.230317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.238939 (+ 8622us) service_pool.cc:227] Handling call
0806 04:07:39.750441 (+1511502us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b5400168354. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:39.750545 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592022400}
W20260806 04:07:39.798524 4808 rpcz_store.cc:269] 0806 04:07:36.869714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:36.949420 (+ 79706us) service_pool.cc:227] Handling call
0806 04:07:39.761171 (+2811751us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400270bd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:39.761325 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4012028288}
W20260806 04:07:39.759810 5029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 127) took 1626 ms (client timeout 1868 ms). Trace:
I20260806 04:07:39.799242 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34040 (request call id 640) took 1369 ms. Trace:
W20260806 04:07:39.813531 5029 rpcz_store.cc:269] 0806 04:07:38.133096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.133325 (+ 229us) service_pool.cc:227] Handling call
0806 04:07:39.759626 (+1626301us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b540032fad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:39.759754 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1916708224}
W20260806 04:07:39.741181 7335 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.553s user 0.004s sys 0.013s
I20260806 04:07:39.814185 5030 rpcz_store.cc:276] 0806 04:07:38.422428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.422595 (+ 167us) service_pool.cc:227] Handling call
0806 04:07:39.790851 (+1368256us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b540032fad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23de89fc6e 00007f23d7f96807 00007f23d7f96792 00007f23d7f966c1 00007f23d7f9663c 00007f23d7f958f2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:39.791701 (+ 850us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1144635264}
I20260806 04:07:39.820259 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.824095 4931 raft_consensus.cc:2379] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.827400 7335 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.830410 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1849) took 1983 ms. Trace:
I20260806 04:07:39.830879 4805 rpcz_store.cc:276] 0806 04:07:37.846960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.985457 (+138497us) service_pool.cc:227] Handling call
0806 04:07:37.985835 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:39.177242 (+1191407us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.830145 (+652903us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540011dd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:39.830342 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2640555264}
I20260806 04:07:39.832527 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.833499 4928 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.833849 7218 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.835023 7335 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.836762 4925 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:39.837280 7267 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:39.837327 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 219) took 1506 ms. Trace:
W20260806 04:07:39.743969 7214 consensus_meta.cc:287] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.717s user 0.014s sys 0.004s
I20260806 04:07:39.838737 4925 rpcz_store.cc:276] 0806 04:07:38.330933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.332014 (+ 1081us) service_pool.cc:227] Handling call
0806 04:07:39.836370 (+1504356us) spinlock_profiling.cc:243] Waited 7.19 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:39.836605 (+ 235us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:39.837125 (+ 520us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.837271 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16525568}
I20260806 04:07:39.832543 5860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 194) took 1882 ms. Trace:
I20260806 04:07:39.838063 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.834566 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.842913 4928 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.733500 7418 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:39.813004 7280 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.760366 7408 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.753511 7182 raft_consensus.cc:3032] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:39.860546 7182 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:39.748220 7308 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:39.868371 4926 raft_consensus.cc:1226] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:39.859380 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.851275 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.792758 7202 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:39.749917 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.845808 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.790179 7425 raft_consensus.cc:3032] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:39.875018 4931 raft_consensus.cc:2379] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.733647 7379 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.866976 7182 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.840050 5860 rpcz_store.cc:276] 0806 04:07:37.949589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.949708 (+ 119us) service_pool.cc:227] Handling call
0806 04:07:38.948552 (+998844us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:38.948757 (+ 205us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:39.125671 (+176914us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:39.125735 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:39.125809 (+ 74us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:39.125841 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.832256 (+706415us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54007e04d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:39.832331 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:39.832473 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448747136}
W20260806 04:07:39.874179 4992 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.016s sys 0.016s
I20260806 04:07:39.746351 7437 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:39.880741 7437 raft_consensus.cc:686] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:39.881979 4992 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.540s user 0.017s sys 0.016s
I20260806 04:07:39.870643 4925 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:39.878189 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:39.882566 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 576550us
I20260806 04:07:39.868438 7280 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.888579 7437 consensus_queue.cc:237] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.890615 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.879980 7202 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.895219 7280 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:39.895960 7202 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.896570 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:39.897320 4925 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.898068 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:39.874755 7308 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:39.872632 7337 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.909467 7379 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:39.909724 7337 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.852869 7214 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:39.895968 7444 log.cc:924] Time spent T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.624s user 0.004s sys 0.001s
I20260806 04:07:39.899994 4924 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:39.882387 7425 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:39.922971 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:39.922930 7425 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:39.923903 4924 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.924136 7425 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:39.923934 5812 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:39.924909 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.925412 7390 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:39.925752 4924 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:39.870041 7408 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:39.911192 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.904616 7001 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:39.906150 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 522) took 1002 ms. Trace:
I20260806 04:07:39.931902 4931 raft_consensus.cc:2379] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:39.934226 4817 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:39.928624 7390 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:39.933544 4924 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:39.964677 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 481768us
W20260806 04:07:39.965306 4993 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.000s sys 0.017s
W20260806 04:07:39.965742 4993 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.000s sys 0.017s
I20260806 04:07:39.965754 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:39.955941 5814 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.045s sys 0.084s
W20260806 04:07:39.973577 5814 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.045s sys 0.084s
W20260806 04:07:39.950055 7447 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.662s user 0.017s sys 0.000s
I20260806 04:07:39.926188 7278 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:39.983171 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.003s sys 0.014s
W20260806 04:07:40.153841 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.003s sys 0.014s
I20260806 04:07:40.003701 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.171870 5860 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:40.021728 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.988273 7447 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:40.173863 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 711216us
I20260806 04:07:39.974444 7202 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:39.974591 7331 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:39.974792 7382 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.985391 7345 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.985777 7337 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:39.990131 4931 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:39.989952 7257 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.033697 4992 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:40.003942 7001 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.983945 7308 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:39.998893 7453 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.011s sys 0.036s
I20260806 04:07:39.974273 7325 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:40.167105 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:39.974998 7316 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:39.971410 7379 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:40.021703 4995 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:40.247030 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:07:40.125097 4813 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.003s sys 0.012s
I20260806 04:07:40.024655 7003 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:40.192648 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 158990us
I20260806 04:07:40.195091 7257 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.196786 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.199468 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.199801 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.200721 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:39.935467 5856 rpcz_store.cc:276] 0806 04:07:38.904084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.916176 (+ 12092us) service_pool.cc:227] Handling call
0806 04:07:39.749799 (+833623us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:39.904579 (+154780us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18336640}
I20260806 04:07:40.202404 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.205318 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.271994 5029 raft_consensus.cc:2362] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2. Re-sending same reply.
W20260806 04:07:40.268074 4879 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.016s sys 0.038s
W20260806 04:07:40.247922 4813 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.895s user 0.003s sys 0.012s
W20260806 04:07:40.250408 7488 log.cc:924] Time spent T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.228s user 0.000s sys 0.000s
W20260806 04:07:40.206211 7453 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.884s user 0.015s sys 0.037s
I20260806 04:07:40.249946 7316 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.251462 7383 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:40.255110 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.255692 4992 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:40.305637 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 50008us
I20260806 04:07:40.307746 4992 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:40.247921 5034 raft_consensus.cc:2362] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2. Re-sending same reply.
I20260806 04:07:40.260372 5036 raft_consensus.cc:2362] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
W20260806 04:07:40.261601 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 239985us
I20260806 04:07:40.255882 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.310824 5038 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:40.209659 6851 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:40.311694 6851 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.314096 6851 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.252180 7379 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.268810 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.209280 4930 raft_consensus.cc:2379] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:40.273983 4879 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.788s user 0.016s sys 0.038s
W20260806 04:07:40.250476 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1501) took 1463 ms (client timeout 1891 ms). Trace:
I20260806 04:07:40.209522 7326 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.298161 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.255795 7199 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.209863 7325 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:40.317015 6851 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:40.192390 7337 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:40.342679 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 857005us
I20260806 04:07:40.324626 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.340082 5038 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
W20260806 04:07:40.343307 5035 rpcz_store.cc:269] 0806 04:07:38.786914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.787141 (+ 227us) service_pool.cc:227] Handling call
0806 04:07:40.250277 (+1463136us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400167e54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:40.250419 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944724224}
I20260806 04:07:40.189236 7331 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.259178 7268 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.188488 7345 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.184947 7382 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.187374 7202 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.280687 7367 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.319144 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.428896 5029 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:40.353250 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.280130 7311 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:40.434543 5037 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:40.434823 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.436285 5866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 488) took 1282 ms. Trace:
I20260806 04:07:40.436776 5866 rpcz_store.cc:276] 0806 04:07:39.154245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.512458 (+358213us) service_pool.cc:227] Handling call
0806 04:07:39.512757 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:39.650936 (+138179us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:39.650980 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:39.651035 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:39.651078 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:40.435990 (+784912us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54005604d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:40.436053 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:40.436215 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":317117696}
I20260806 04:07:40.338140 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.361481 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.326622 7408 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.322695 5857 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:40.197548 7001 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:40.410248 4770 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.030s sys 0.059s
W20260806 04:07:40.450680 4770 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.573s user 0.031s sys 0.059s
I20260806 04:07:40.450697 5032 raft_consensus.cc:2362] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3. Re-sending same reply.
I20260806 04:07:40.409466 6851 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.297590 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.319831 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.323335 7257 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:40.462982 7257 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.463341 6851 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.463512 7257 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.463896 6851 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:40.463958 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.426980 4876 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:40.430194 4930 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:40.464815 4919 raft_consensus.cc:2379] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:40.465407 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:40.465654 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.465972 4930 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.466207 4876 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:40.447120 7099 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.426869 5853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 204) took 1048 ms. Trace:
I20260806 04:07:40.323918 7214 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.326114 7278 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.292898 7487 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:40.308625 7313 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:40.355423 7288 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:40.345275 7204 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:40.408665 4931 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:40.449920 7408 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.321069 7316 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:40.416764 7498 log.cc:924] Time spent T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260806 04:07:40.416690 7325 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:40.451227 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 574156us
I20260806 04:07:40.459775 5029 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:40.197410 7390 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.441923 5026 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:40.464452 6851 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.318519 7492 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.205s
I20260806 04:07:40.380296 7199 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.372548 7326 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.444693 7367 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.491935 7288 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:40.446688 5867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 200) took 1511 ms. Trace:
I20260806 04:07:40.466272 4919 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:40.429924 7268 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.541762 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.539934 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:40.547178 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 193882us
I20260806 04:07:40.550057 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.550691 5026 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:40.540037 7447 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.543550 7492 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:40.514283 7313 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.563139 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.563356 7492 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.563938 5029 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:40.565562 7313 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.563139 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:40.568190 7503 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260806 04:07:40.568709 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.569383 5037 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:40.379408 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.570695 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.541810 7288 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.583714 5029 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:40.545198 7485 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.603258 5026 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:40.598317 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.611510 7345 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.611522 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1865) took 1379 ms. Trace:
I20260806 04:07:40.616135 4804 rpcz_store.cc:276] 0806 04:07:39.232577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.234133 (+ 1556us) service_pool.cc:227] Handling call
0806 04:07:39.234572 (+ 439us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:39.600808 (+366236us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:40.611125 (+1010317us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400380254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:40.611438 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":777977088}
W20260806 04:07:40.506271 7371 consensus_meta.cc:287] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.939s user 0.014s sys 0.001s
I20260806 04:07:40.469933 5853 rpcz_store.cc:276] 0806 04:07:39.369772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.823666 (+453894us) service_pool.cc:227] Handling call
0806 04:07:39.824266 (+ 600us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:39.824633 (+ 367us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:07:39.989365 (+164732us) log.cc:842] Serialized 14 byte log entry
0806 04:07:40.297006 (+307641us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:40.302411 (+ 5405us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:40.344935 (+ 42524us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:40.374905 (+ 29970us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:40.418059 (+ 43154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83723,"prepare.queue_time_us":206488,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":96843,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":23763,"thread_start_us":314373,"threads_started":2}
I20260806 04:07:40.617949 7371 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:40.619895 7445 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 1.516s user 0.008s sys 0.010s
I20260806 04:07:40.467003 7099 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.621834 4924 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:40.489348 7214 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:40.622692 7030 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:40.491569 7204 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:40.494391 7278 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:40.631834 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1165) took 2372 ms. Trace:
I20260806 04:07:40.636426 7371 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.531855 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.531440 5867 rpcz_store.cc:276] 0806 04:07:38.924918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.930631 (+ 5713us) service_pool.cc:227] Handling call
0806 04:07:40.434617 (+1503986us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:40.435747 (+ 1130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2933632}
I20260806 04:07:40.500216 7423 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:40.347273 7383 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.534049 7193 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:40.534294 7326 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.536073 7199 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.537837 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:40.538503 7349 consensus_meta.cc:287] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.985s user 0.008s sys 0.007s
I20260806 04:07:40.539427 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.541106 6851 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.408943 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.632308 4810 rpcz_store.cc:276] 0806 04:07:38.252270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.252478 (+ 208us) service_pool.cc:227] Handling call
0806 04:07:38.252970 (+ 492us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:38.256420 (+ 3450us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:38.256479 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:38.264709 (+ 8230us) log.cc:842] Serialized 26 byte log entry
0806 04:07:38.948364 (+683655us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:39.296037 (+347673us) log.cc:842] Serialized 14 byte log entry
0806 04:07:39.296660 (+ 623us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:39.296817 (+ 157us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:40.618930 (+1322113us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:40.624241 (+ 5311us) raft_consensus.cc:1671] finished
0806 04:07:40.624334 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:40.624492 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1186,"prepare.queue_time_us":384521,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":1597,"raft.run_cpu_time_us":547,"raft.run_wall_time_us":34248,"spinlock_wait_cycles":3515264,"thread_start_us":942516,"threads_started":2,"wal-append.queue_time_us":708540}
I20260806 04:07:40.505337 7408 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.566365 7492 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.505716 7484 consensus_queue.cc:1046] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 04:07:40.640204 7409 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.589641 7331 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.591910 7337 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.593902 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.641772 7423 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:40.623164 7214 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.605173 7001 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.606732 7379 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:40.608181 7382 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:40.609831 7485 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.616864 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1466) took 1826 ms. Trace:
W20260806 04:07:40.484448 7283 consensus_meta.cc:287] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.949s user 0.005s sys 0.012s
I20260806 04:07:40.477229 5860 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
W20260806 04:07:40.612573 5862 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 193) took 2663 ms (client timeout 2128 ms). Trace:
I20260806 04:07:40.627465 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.614519 5032 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:40.719121 5026 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.719668 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.720438 5032 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.675836 7437 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.638024 5038 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:40.573092 7316 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.665680 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.723270 5036 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:40.718837 5862 rpcz_store.cc:269] 0806 04:07:37.949299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:37.949432 (+ 133us) service_pool.cc:227] Handling call
0806 04:07:38.342581 (+393149us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:40.612253 (+2269672us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b540051ed54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:40.612411 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1303304704}
I20260806 04:07:40.539996 7367 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:40.668054 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 201903us
I20260806 04:07:40.668090 4879 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:40.725672 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 57666.7us
I20260806 04:07:40.634491 7204 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.588713 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.730921 5037 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:40.613843 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 489) took 1388 ms. Trace:
I20260806 04:07:40.630753 7497 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.610597 7202 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.600698 7419 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:40.600909 7268 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.660988 7278 raft_consensus.cc:686] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:40.670126 7349 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.670737 7418 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.671576 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.688841 4923 rpcz_store.cc:276] 0806 04:07:38.790249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.790499 (+ 250us) service_pool.cc:227] Handling call
0806 04:07:38.790880 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:38.791175 (+ 295us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:07:39.481315 (+690140us) log.cc:842] Serialized 14 byte log entry
0806 04:07:40.616234 (+1134919us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b18000fd240. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e157b
0806 04:07:40.616467 (+ 233us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:40.616527 (+ 60us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:40.616584 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:40.616680 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:40.616815 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":582182,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":1351,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":9679104,"thread_start_us":1150943,"threads_started":2,"wal-append.queue_time_us":464965}
I20260806 04:07:40.725628 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:40.512703 4930 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:40.715457 7311 raft_consensus.cc:686] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:40.727605 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.660162 7193 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.716216 5856 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:40.743391 5026 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:40.748693 5035 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:40.749727 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:40.753593 7467 consensus_meta.cc:287] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.040s user 0.007s sys 0.009s
I20260806 04:07:40.774993 4770 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:40.777093 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.732576 5859 rpcz_store.cc:276] 0806 04:07:39.225276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.382950 (+157674us) service_pool.cc:227] Handling call
0806 04:07:40.613613 (+1230663us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400770954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:40.613770 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":447058176}
I20260806 04:07:40.782191 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.790544 5026 raft_consensus.cc:2379] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 2: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:40.777835 5035 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:40.791659 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.792665 5026 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:40.797449 7485 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.727205 7283 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.729763 7437 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.757048 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.670377 7184 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:40.683832 7409 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.759770 5036 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:40.770138 5029 raft_consensus.cc:2379] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:40.730675 7382 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:40.771708 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.771688 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.755877 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.780475 5863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 203) took 1812 ms. Trace:
I20260806 04:07:40.728981 7316 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.767540 7418 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.861625 4768 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:40.812706 7504 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
W20260806 04:07:40.813473 7495 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260806 04:07:40.813880 7516 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:40.814585 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.816442 5866 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:40.824319 7278 consensus_queue.cc:237] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.826169 7408 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.826879 7396 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:40.828822 7193 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.780861 7515 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:40.830338 7383 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.750686 7497 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:40.831369 7425 consensus_meta.cc:287] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.907s user 0.003s sys 0.012s
I20260806 04:07:40.832051 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.879638 7516 raft_consensus.cc:979] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:40.843505 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.870121 7521 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:40.848693 7325 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.789712 7367 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:40.907951 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 210) took 2012 ms. Trace:
I20260806 04:07:40.853588 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.857952 5812 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:40.857509 5863 rpcz_store.cc:276] 0806 04:07:38.967737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.968048 (+ 311us) service_pool.cc:227] Handling call
0806 04:07:40.779581 (+1811533us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:40.779840 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:40.780067 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:40.780102 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:40.780158 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:40.780200 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:40.780267 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:40.780403 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2302208}
I20260806 04:07:40.909349 5036 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:07:40.909617 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51747.7us
I20260806 04:07:40.860160 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:40.910441 4930 raft_consensus.cc:2379] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:40.910292 5812 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:40.911242 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.911422 5812 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:40.912300 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.861076 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.913151 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:40.863950 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:40.913906 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 50094.8us
W20260806 04:07:40.865242 4995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.030s sys 0.055s
I20260806 04:07:40.865356 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:40.915042 4995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.030s sys 0.056s
I20260806 04:07:40.915234 5029 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:40.810417 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:40.781623 4919 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:40.874487 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:40.784543 7467 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.737552 5038 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:40.798635 7311 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.888422 5032 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
W20260806 04:07:40.894311 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 167627us
I20260806 04:07:40.822559 7204 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.896692 5037 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:40.845460 7349 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.848299 7257 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:40.860077 7382 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.928102 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.900609 7283 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.900808 7437 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:40.856807 7184 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:40.863108 4927 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:40.935770 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:40.907389 4931 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:40.917246 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.917948 7308 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:40.918390 7193 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:40.918503 7205 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:40.938949 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:40.918998 7325 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:40.783049 7423 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:40.920285 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:40.913319 4923 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:41.083303 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1371) took 1110 ms. Trace:
I20260806 04:07:41.083740 4923 rpcz_store.cc:276] 0806 04:07:39.972854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.860771 (+887917us) service_pool.cc:227] Handling call
0806 04:07:41.083241 (+222470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:40.908335 4813 rpcz_store.cc:276] 0806 04:07:38.895368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.895502 (+ 134us) service_pool.cc:227] Handling call
0806 04:07:38.895950 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:39.292853 (+396903us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:39.296924 (+ 4071us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:39.316160 (+ 19236us) log.cc:842] Serialized 26 byte log entry
0806 04:07:40.273658 (+957498us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:40.273707 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:40.273783 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:40.907708 (+633925us) raft_consensus.cc:1671] finished
0806 04:07:40.907731 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:40.907880 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":59381,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":217,"thread_start_us":1316676,"threads_started":2,"wal-append.queue_time_us":1246528}
I20260806 04:07:40.923002 5026 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:40.923195 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.923370 5812 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:40.924520 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.897246 7396 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:40.900116 7214 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:40.933177 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 207672us
I20260806 04:07:40.934018 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.086056 5032 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:41.086169 4876 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:40.901641 7521 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:40.901230 7497 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:41.091341 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 168017us
I20260806 04:07:41.091509 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:41.092098 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 171872us
I20260806 04:07:40.890328 7268 consensus_queue.cc:1046] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260806 04:07:40.936434 5029 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.937428 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.097465 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.919541 7515 raft_consensus.cc:1067] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer d7c76b6dcd8945cd818b8107caf4a174 to start an election
I20260806 04:07:41.098402 5032 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:41.099571 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.099900 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.101008 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.102425 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:40.939716 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:40.989013 7409 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:41.048389 4769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.664s user 0.008s sys 0.013s
I20260806 04:07:41.034137 7382 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.060478 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1355) took 1253 ms. Trace:
W20260806 04:07:41.105274 4769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.721s user 0.009s sys 0.014s
I20260806 04:07:41.106943 4923 raft_consensus.cc:2379] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:41.100776 5038 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:41.108218 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:41.016987 7525 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260806 04:07:41.034773 7316 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:41.034919 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1356) took 1228 ms. Trace:
I20260806 04:07:41.111284 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.111415 4924 rpcz_store.cc:276] 0806 04:07:39.806577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.952793 (+146216us) service_pool.cc:227] Handling call
0806 04:07:40.621609 (+668816us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:40.622362 (+ 753us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.034615 (+412253us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54000131d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:41.034841 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1493523840}
I20260806 04:07:41.112553 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.033529 7184 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.035178 7418 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.050417 5860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 491) took 1049 ms. Trace:
I20260806 04:07:41.113436 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.109644 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:40.987254 7205 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:41.056208 7511 raft_consensus.cc:3032] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:40.991432 7467 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.052639 7461 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:40.925287 7509 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 04:07:41.145783 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 134567us
W20260806 04:07:41.045001 7506 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.530s user 0.002s sys 0.000s
I20260806 04:07:40.938890 7308 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:40.944988 5853 raft_consensus.cc:2354] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3.
I20260806 04:07:40.907481 7367 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:41.054404 7280 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.113950 7476 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 04:07:41.068991 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 212) took 2168 ms (client timeout 2094 ms). Trace:
I20260806 04:07:41.056283 7288 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.038754 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1346) took 1235 ms. Trace:
I20260806 04:07:40.982036 5028 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:41.075608 7099 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.075896 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.166177 5856 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:41.167066 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:41.168386 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.168993 5034 raft_consensus.cc:2379] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:41.169682 5856 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:41.170542 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:41.080169 7003 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.081393 7429 raft_consensus.cc:979] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:40.988425 7318 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.171164 5034 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:40.997546 5037 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:41.171429 7429 raft_consensus.cc:1067] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:07:41.077558 7182 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.172267 5037 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:40.986968 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.173574 5859 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:40.920640 7383 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:41.093048 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:41.174598 7383 raft_consensus.cc:686] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:41.087687 4876 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:41.104826 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:41.176095 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 88519.6us
I20260806 04:07:41.176258 7410 raft_consensus.cc:3032] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:41.175964 7383 consensus_queue.cc:237] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.176692 7410 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:41.177052 4876 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:41.095510 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 69015.4us
I20260806 04:07:41.105439 4928 rpcz_store.cc:276] 0806 04:07:39.806269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.852409 (+ 46140us) service_pool.cc:227] Handling call
0806 04:07:41.059413 (+1207004us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54003552d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:41.059604 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707547392}
I20260806 04:07:41.143549 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.178273 5036 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:41.179447 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:41.178251 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 85319.4us
I20260806 04:07:41.164940 4926 rpcz_store.cc:276] 0806 04:07:39.802992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.803128 (+ 136us) service_pool.cc:227] Handling call
0806 04:07:39.828563 (+ 25435us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:39.872312 (+ 43749us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.038427 (+1166115us) spinlock_profiling.cc:243] Waited 36.4 ms on lock 0x7b54004137d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:41.038675 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83621760}
I20260806 04:07:41.182024 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.143217 4930 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:41.183236 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.146945 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:41.183871 4813 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:41.184320 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.184726 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.144016 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.185164 4813 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:41.150628 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.091594 7521 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.104492 7214 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:41.057796 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.110714 7474 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:41.145583 7511 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:41.145798 7378 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.152237 7308 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:41.157236 4817 rpcz_store.cc:269] 0806 04:07:38.900255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:38.900383 (+ 128us) service_pool.cc:227] Handling call
0806 04:07:41.068768 (+2168385us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003cf0d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:41.068920 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2335437184}
I20260806 04:07:41.159228 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:41.159643 7280 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.160406 7476 raft_consensus.cc:686] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:41.200403 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.201603 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.201264 7476 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.160152 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.210173 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:41.161069 7326 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:41.162405 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.145596 7479 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:41.163204 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.163239 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.211983 4814 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:41.163432 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.143534 4924 raft_consensus.cc:2379] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:41.186040 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.183933 5866 raft_consensus.cc:2362] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3. Re-sending same reply.
I20260806 04:07:41.096483 7396 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.149315 7219 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.213145 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.152788 7325 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.153053 7371 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.155151 7367 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.221509 4813 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.189229 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.189757 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.186365 5038 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:41.187872 7214 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.227468 5038 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:41.191831 5029 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:41.226933 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:41.180042 4804 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:41.228905 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.154800 7349 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.193567 4876 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:41.185117 4930 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:41.233286 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:41.233491 4876 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:41.233804 4930 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.234561 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.199079 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.187747 7429 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:41.208106 7378 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.165280 7288 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:41.209564 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 50421.3us
I20260806 04:07:41.242841 5812 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:41.167107 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.096767 7407 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:41.214633 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.215515 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.211179 5034 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:41.244206 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.217628 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.181918 7485 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:07:41.219424 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.220381 4816 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:07:41.209656 4919 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:41.224004 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.224880 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.235072 7214 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:41.148430 7409 consensus_queue.cc:1046] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 04:07:41.230335 4923 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.235392 4817 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.235356 4876 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.264459 4876 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:41.264678 4879 leader_election.cc:398] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3.
I20260806 04:07:41.265573 4879 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:41.239923 5032 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:41.193806 7515 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:41.243521 5035 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:41.239336 7429 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:41.242214 7030 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:41.218467 7193 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:41.246697 5038 raft_consensus.cc:1261] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:41.247630 7390 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.250137 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.239645 7268 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:41.277201 5035 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:41.287415 4810 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.250521 5863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:45574
I20260806 04:07:41.250584 5812 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:41.288230 5863 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:41.288597 5863 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:41.115381 5860 rpcz_store.cc:276] 0806 04:07:40.001244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.172848 (+171604us) service_pool.cc:227] Handling call
0806 04:07:41.050197 (+877349us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540042d1d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:41.050337 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":496830336}
I20260806 04:07:41.252367 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.289387 4809 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:41.252278 7510 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:41.293422 4992 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.209232 7099 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260806 04:07:41.257254 4924 raft_consensus.cc:2354] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:07:41.225922 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.260603 4818 raft_consensus.cc:1226] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:41.236994 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.245098 7425 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:41.273574 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.289707 4995 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:41.255384 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.291949 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1376) took 1304 ms. Trace:
I20260806 04:07:41.221686 7326 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.263527 7511 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.326282 4923 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:41.223495 7479 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.221143 7219 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.301313 4930 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:41.321702 4928 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:41.321593 4931 rpcz_store.cc:276] 0806 04:07:39.987357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.500168 (+512811us) service_pool.cc:227] Handling call
0806 04:07:41.291804 (+791636us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b540006e854. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:41.291900 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555456384}
W20260806 04:07:41.325173 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51728us
I20260806 04:07:41.361538 4768 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.362957 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:41.260450 7418 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:41.296023 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.370108 5038 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:41.296579 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.284001 7429 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.370999 5035 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:41.371235 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:41.372298 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.262950 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1468) took 1287 ms. Trace:
I20260806 04:07:41.354964 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.242020 7349 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.328231 7199 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:41.286597 7193 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.189678 7410 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.225780 7325 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.236248 7371 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.266446 7485 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:41.268011 7184 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:41.353504 4817 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:41.409540 4804 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:41.222050 7474 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.366336 4816 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:41.413508 4816 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.367065 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.367739 4920 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:41.369060 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.283126 7407 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:41.302361 7484 raft_consensus.cc:2741] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:07:41.375558 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.416945 5038 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:41.376472 4930 raft_consensus.cc:2379] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:41.418339 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 140795us
I20260806 04:07:41.306119 7510 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.298995 7268 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.304733 7409 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:41.384985 4924 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:41.286271 7515 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.351028 7218 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:41.385633 4919 rpcz_store.cc:276] 0806 04:07:39.975395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.944817 (+969422us) service_pool.cc:227] Handling call
0806 04:07:41.259065 (+314248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:41.404482 7485 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.300706 7389 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:41.351179 7316 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:41.386024 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.472077 4931 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:41.374218 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.389102 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.473209 4928 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:41.291062 7278 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260806 04:07:41.328898 7528 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.394340 5037 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:41.393255 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1779) took 1128 ms. Trace:
I20260806 04:07:41.404502 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 239) took 1655 ms. Trace:
I20260806 04:07:41.385080 7511 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.475618 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.477586 7479 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.471458 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:41.479585 7349 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.462419 7099 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.480342 7099 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.482327 7099 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.483531 7219 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.484143 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.423846 7184 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.485204 7030 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.485711 7030 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:41.424269 7419 consensus_queue.cc:1159] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7c76b6dcd8945cd818b8107caf4a174" responder_term: 4 status { last_received { term: 2 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:07:41.371316 7204 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:41.418671 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1379) took 1428 ms. Trace:
I20260806 04:07:41.418346 5038 raft_consensus.cc:1261] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:41.424639 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.376673 7497 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:41.300362 7390 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.242597 7205 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.429028 4805 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:41.431708 4814 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.427590 7268 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:41.440153 4816 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.453248 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:41.456391 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.495488 4930 rpcz_store.cc:276] 0806 04:07:39.990261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.371002 (+1380741us) service_pool.cc:227] Handling call
0806 04:07:41.418595 (+ 47593us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:41.503963 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.373070 5035 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:41.495838 4926 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.394397 7193 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.384830 7522 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:41.496320 4923 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
W20260806 04:07:41.511543 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 143311us
I20260806 04:07:41.316422 7315 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.428355 7510 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.475338 4927 rpcz_store.cc:276] 0806 04:07:39.748747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.748916 (+ 169us) service_pool.cc:227] Handling call
0806 04:07:39.749245 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:40.936236 (+1186991us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.404239 (+468003us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540018fbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:41.404441 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2561106176}
I20260806 04:07:41.475072 5028 rpcz_store.cc:276] 0806 04:07:40.264759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.297798 (+ 33039us) service_pool.cc:227] Handling call
0806 04:07:41.393058 (+1095260us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b540032f0d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:41.393197 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1372003200}
I20260806 04:07:41.499953 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.540042 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.541405 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.542407 4817 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.405318 7199 raft_consensus.cc:686] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:41.501071 4809 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:41.543910 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.544863 4809 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.492213 4925 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
W20260806 04:07:41.503159 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 50145.9us
I20260806 04:07:41.551257 4926 raft_consensus.cc:2396] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:41.495296 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1469) took 1505 ms. Trace:
I20260806 04:07:41.496376 7030 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.490455 7389 raft_consensus.cc:686] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:41.471800 7316 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:41.531932 7325 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.523643 5863 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.554788 4928 rpcz_store.cc:276] 0806 04:07:39.989904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.473837 (+1483933us) service_pool.cc:227] Handling call
0806 04:07:41.474131 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.494975 (+ 20844us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.495005 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:41.495046 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:41.495076 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.495138 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:41.495244 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20479,"raft.queue_time_us":27421}
I20260806 04:07:41.597136 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.471508 7485 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.491220 7437 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.468498 7418 raft_consensus.cc:686] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:41.492288 7410 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.668498 4931 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:41.668561 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.534510 7479 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:41.669627 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.489028 7419 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:07:41.611413 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 63569.7us
I20260806 04:07:41.486644 7184 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.678369 5863 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.678277 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.692260 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.692724 4809 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.487856 7474 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.494692 7326 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.689007 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.701164 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.496484 7497 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.494015 7396 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.685416 5857 raft_consensus.cc:2379] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:41.499045 7003 consensus_queue.cc:1159] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "99690415a7b3441fbdb5e943667556a4" responder_term: 4 status { last_received { term: 2 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:07:41.495272 7378 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:41.698626 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 324479us
I20260806 04:07:41.626425 7316 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.612116 7425 raft_consensus.cc:686] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:41.702896 5863 raft_consensus.cc:1226] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:41.490202 7515 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.611287 7182 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.675160 7516 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.712239 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.713909 5859 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:41.615157 7557 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:41.672303 7390 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:41.722481 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.753582 4809 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:41.679862 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.723735 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.707015 6970 raft_consensus.cc:3032] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:41.743651 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:41.760941 5030 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:41.678020 7099 consensus_queue.cc:1046] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 04:07:41.700914 7407 raft_consensus.cc:686] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:41.715174 7423 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.746124 4931 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:41.486613 7308 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.706969 7497 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:41.718165 7184 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:41.725860 7409 raft_consensus.cc:686] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:41.693352 7437 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.761840 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.696116 7419 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.768543 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:41.778716 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.778719 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.769696 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:41.783233 7099 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:41.783795 7099 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:41.783125 7581 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:41.784739 7581 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.785151 7581 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:41.789356 7326 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:41.789983 4768 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:41.792493 7429 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.770227 7378 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:41.769804 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1470) took 1776 ms. Trace:
I20260806 04:07:41.759236 6970 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:41.745994 7199 consensus_queue.cc:237] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.750046 7182 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.784226 7099 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:41.795814 7419 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:41.806047 7367 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.568s user 0.005s sys 0.005s
I20260806 04:07:41.760161 7515 consensus_queue.cc:1046] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260806 04:07:41.775530 7390 raft_consensus.cc:3032] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:41.810853 7349 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:41.813071 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.768240 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.793713 7425 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.785197 7407 consensus_queue.cc:237] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.811501 4928 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:41.753659 7516 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.875339 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:41.804630 4876 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.895241 5863 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:07:41.899655 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 95166.5us
I20260806 04:07:41.831506 7308 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.831657 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:41.832347 4927 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:41.749276 7389 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:41.900667 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:41.907238 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:41.893399 7419 raft_consensus.cc:3032] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:41.776023 7423 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.804982 7326 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.778095 7522 consensus_queue.cc:1046] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260806 04:07:41.918622 7423 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.763144 7418 consensus_queue.cc:237] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:41.818789 7521 consensus_meta.cc:287] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.623s user 0.011s sys 0.000s
I20260806 04:07:41.920157 7521 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.779822 7184 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.922361 7521 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.923439 7184 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.923696 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:41.902691 7214 consensus_meta.cc:287] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.618s user 0.007s sys 0.003s
I20260806 04:07:41.926313 7214 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:41.817257 7564 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:41.878784 7390 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:41.915850 7280 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.811007 4919 rpcz_store.cc:276] 0806 04:07:39.991317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.502498 (+1511181us) service_pool.cc:227] Handling call
0806 04:07:41.704364 (+201866us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.724001 (+ 19637us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.730236 (+ 6235us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:41.740777 (+ 10541us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:41.745531 (+ 4754us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.752916 (+ 7385us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:41.767706 (+ 14790us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:41.818153 7371 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:41.805187 7325 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:41.822149 7219 consensus_queue.cc:1046] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260806 04:07:41.879323 7367 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:41.980916 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 80300.9us
W20260806 04:07:41.967518 7517 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.034s user 0.074s sys 0.081s
I20260806 04:07:41.909821 7515 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:41.961175 7326 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:41.931883 7419 raft_consensus.cc:729] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:41.961014 7564 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:07:42.013514 7517 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.079s user 0.075s sys 0.081s
I20260806 04:07:41.978660 7214 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:41.994460 7184 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:42.028463 4813 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:42.010468 7515 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.010805 7429 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.009033 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:42.038524 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 232842us
I20260806 04:07:42.039382 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:41.831017 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:42.047206 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:42.023515 7564 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:42.028469 7184 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:41.994792 7556 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260806 04:07:41.909541 7409 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.067056 5028 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:42.004351 7325 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.068107 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:42.080327 5030 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:42.041535 7515 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.069130 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:42.078325 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:42.078457 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:42.081367 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:42.075011 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.091243 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:42.091915 5863 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:42.093070 4879 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:42.084817 5028 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:42.058748 7219 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:42.056182 7184 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:42.109413 4876 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.113523 7567 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260806 04:07:42.120369 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1510) took 1522 ms. Trace:
I20260806 04:07:42.120738 7567 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:42.121490 7567 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.121930 7182 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:42.122822 4925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1366) took 2152 ms (client timeout 1508 ms). Trace:
W20260806 04:07:42.123241 4925 rpcz_store.cc:269] 0806 04:07:39.970304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.971417 (+ 1113us) service_pool.cc:227] Handling call
0806 04:07:42.122578 (+2151161us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54003557d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.122722 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3033169792}
W20260806 04:07:42.123873 7530 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.974s user 0.005s sys 0.000s
W20260806 04:07:42.124118 7511 consensus_meta.cc:287] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.648s user 0.012s sys 0.002s
I20260806 04:07:42.110746 7219 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.124816 7219 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:42.125208 5035 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:42.125864 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.127146 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:42.128183 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.129137 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.129822 4818 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:42.130712 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.131834 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.132400 4928 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:42.130494 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.124533 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:42.079653 7573 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:42.067085 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.135106 5812 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:42.141064 5812 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:42.080053 7378 raft_consensus.cc:686] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:42.144337 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.101101 7437 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:42.147348 4876 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.120898 5037 rpcz_store.cc:276] 0806 04:07:40.599090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.172942 (+573852us) service_pool.cc:227] Handling call
0806 04:07:41.394022 (+221080us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:41.474735 (+ 80713us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.120114 (+645379us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b540032f0d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.120297 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":507677568}
I20260806 04:07:42.153266 4876 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.069887 7521 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260806 04:07:42.154202 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.157869 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:42.121973 7567 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:42.138589 7349 raft_consensus.cc:686] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:42.155624 7461 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:42.164821 7218 consensus_meta.cc:287] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.703s user 0.015s sys 0.000s
I20260806 04:07:42.166210 7485 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.151631 7335 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:07:42.156656 7315 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.624s user 0.010s sys 0.010s
I20260806 04:07:42.139226 7573 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.068423 7390 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.167348 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.285929 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:42.287737 7560 log.cc:924] Time spent T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
W20260806 04:07:42.288695 7539 log.cc:924] Time spent T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.813s user 0.000s sys 0.002s
I20260806 04:07:42.290112 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1511) took 1689 ms. Trace:
I20260806 04:07:42.291534 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.290539 5036 rpcz_store.cc:276] 0806 04:07:40.602383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.179038 (+576655us) service_pool.cc:227] Handling call
0806 04:07:41.200992 (+ 21954us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:41.201338 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.201364 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:41.201817 (+ 453us) log.cc:842] Serialized 26 byte log entry
0806 04:07:41.377095 (+175278us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:41.751285 (+374190us) log.cc:842] Serialized 14 byte log entry
0806 04:07:41.751680 (+ 395us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.751767 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:42.287974 (+536207us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:42.289697 (+ 1723us) raft_consensus.cc:1671] finished
0806 04:07:42.289829 (+ 132us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:42.290027 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3170,"prepare.queue_time_us":334224,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":6217,"raft.queue_time_us":10652,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":8050,"spinlock_wait_cycles":141254656,"thread_start_us":508968,"threads_started":2,"wal-append.queue_time_us":266779}
I20260806 04:07:42.292240 4919 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:42.293413 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:42.311326 4919 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:42.312225 7378 consensus_queue.cc:237] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.314113 7325 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:42.319981 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:42.325632 5863 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:42.326855 4770 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:42.259032 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.328924 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:42.326403 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 149) took 1441 ms. Trace:
I20260806 04:07:42.346141 5032 rpcz_store.cc:276] 0806 04:07:40.884543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.100717 (+216174us) service_pool.cc:227] Handling call
0806 04:07:42.325672 (+1224955us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b540037df54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.326341 (+ 669us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":317542272}
I20260806 04:07:42.351125 5867 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:42.350106 7511 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:42.355577 7509 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:42.243829 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 90584.1us
I20260806 04:07:42.381460 4996 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:42.387557 4996 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:42.142858 7367 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:42.391037 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 293840us
I20260806 04:07:42.398561 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.152855 7308 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:42.344842 7335 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.350843 7573 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:42.363665 7511 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.429292 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1871) took 1269 ms. Trace:
W20260806 04:07:42.249514 7193 consensus_meta.cc:287] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.744s user 0.006s sys 0.011s
I20260806 04:07:42.159808 7572 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:42.352399 7267 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:42.411568 7349 consensus_queue.cc:237] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.415452 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.419783 7182 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:42.422327 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 1.12793e+06us
I20260806 04:07:42.445912 4992 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.448172 4992 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.350821 7587 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:42.408432 7308 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:42.410238 4768 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:42.348033 7461 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.212275 7583 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:42.429687 4805 rpcz_store.cc:276] 0806 04:07:41.160312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.162750 (+ 2438us) service_pool.cc:227] Handling call
0806 04:07:41.163047 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.501801 (+338754us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.429045 (+927244us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54002a44d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:42.429231 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":610771712}
I20260806 04:07:42.363739 7509 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.210902 7516 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.233294 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:42.436861 4769 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items.
I20260806 04:07:42.365638 7556 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:42.438123 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.407392 7390 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.397239 7396 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:42.425899 7335 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:42.453315 4920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1353) took 2647 ms (client timeout 1908 ms). Trace:
W20260806 04:07:42.423260 5857 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 537) took 1630 ms (client timeout 1610 ms). Trace:
I20260806 04:07:42.454000 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1508) took 1866 ms. Trace:
I20260806 04:07:42.453464 7588 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:42.472067 4920 rpcz_store.cc:269] 0806 04:07:39.805553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.853266 (+ 47713us) service_pool.cc:227] Handling call
0806 04:07:42.453135 (+2599869us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540004fbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.453250 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3038975744}
W20260806 04:07:42.477962 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 75978.7us
I20260806 04:07:42.478403 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.245579 7581 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.472335 5026 rpcz_store.cc:276] 0806 04:07:40.587426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.814568 (+227142us) service_pool.cc:227] Handling call
0806 04:07:40.922816 (+108248us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.090976 (+168160us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.453788 (+1362812us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000bd950. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.453937 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248136064}
I20260806 04:07:42.455056 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:42.462992 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.443203 7572 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.472057 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1210)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1902)
W20260806 04:07:42.456612 7528 consensus_meta.cc:287] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.982s user 0.011s sys 0.003s
I20260806 04:07:42.463949 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:42.481069 7528 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:42.439015 7383 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.945s user 0.008s sys 0.011s
I20260806 04:07:42.480413 7581 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.481901 4923 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:42.480456 7572 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:42.481902 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.472759 7556 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:42.477489 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 118327us
I20260806 04:07:42.460060 7308 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:42.490077 7603 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:42.490703 7603 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:42.491108 7588 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:42.491182 7603 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:42.462261 7583 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.483024 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1210)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 241)
W20260806 04:07:42.462450 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 76000.6us
I20260806 04:07:42.450259 7587 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:42.523501 7587 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.488830 4768 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:42.450645 7267 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.456792 7461 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.516662 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:42.536911 5037 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:42.443035 7193 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:42.484632 7383 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.991s user 0.008s sys 0.012s
I20260806 04:07:42.524364 7485 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.541867 7556 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.494385 7396 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:07:42.486321 5857 rpcz_store.cc:269] 0806 04:07:40.792394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.794685 (+ 2291us) service_pool.cc:227] Handling call
0806 04:07:41.483954 (+689269us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:42.423037 (+939083us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b540051ed54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.423190 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45877504}
I20260806 04:07:42.583742 7396 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:42.584240 7396 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:42.525610 7583 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:42.548189 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:42.616228 5863 raft_consensus.cc:2379] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:42.557518 5854 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:42.536414 7593 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:42.616550 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 127823us
I20260806 04:07:42.493420 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:42.643267 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 149925us
I20260806 04:07:42.644588 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:42.645419 5037 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:07:42.617830 4770 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:42.547456 7283 raft_consensus.cc:3032] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 04:07:42.617565 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 169468us
W20260806 04:07:42.622361 7326 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.609s user 0.018s sys 0.009s
W20260806 04:07:42.622637 7099 consensus_meta.cc:287] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.741s user 0.010s sys 0.009s
W20260806 04:07:42.624615 7268 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.028s user 0.013s sys 0.007s
I20260806 04:07:42.628415 7588 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:42.631493 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.633517 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1474) took 2041 ms. Trace:
I20260806 04:07:42.654942 4923 rpcz_store.cc:276] 0806 04:07:40.592216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.511382 (+919166us) service_pool.cc:227] Handling call
0806 04:07:41.511657 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:42.482299 (+970642us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.633250 (+150951us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b540004a454. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:42.633456 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2230863104}
I20260806 04:07:42.657605 5030 raft_consensus.cc:1226] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:42.633054 7572 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.636205 7308 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.670898 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.636878 7509 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:42.611471 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1194)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1902)
I20260806 04:07:42.679575 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1902)
I20260806 04:07:42.637761 7557 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.645272 4882 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:42.648188 4992 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.650382 4768 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:42.646432 7283 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:42.650908 7556 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:07:42.622215 7497 consensus_meta.cc:287] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.826s user 0.011s sys 0.021s
I20260806 04:07:42.651777 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:42.653031 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:42.659265 7511 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:42.684986 7510 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.052s user 0.017s sys 0.005s
I20260806 04:07:42.687873 7378 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:42.656019 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:42.630560 7567 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.675673 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.699779 4809 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:42.671909 7308 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.619589 7193 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.680994 5037 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:42.651793 7345 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.686542 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:42.687633 4920 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:42.667481 7099 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:42.696880 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 51709.9us
I20260806 04:07:42.686318 7557 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.690162 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33548 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
I20260806 04:07:42.653851 4996 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:42.698458 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:42.710952 4923 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:42.710978 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 57236.5us
W20260806 04:07:42.711304 5816 consensus_peers.cc:487] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 -> Peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Couldn't send request to peer df2f1e7811114b8a995f6771fb6a6c8a. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:42.651551 7326 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.702966 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:42.708642 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:42.712416 4996 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:42.706946 7396 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.673207 7268 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.716624 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:42.718261 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 247)
I20260806 04:07:42.721956 4809 raft_consensus.cc:2379] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:42.717775 4876 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.718206 4768 leader_election.cc:398] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
W20260806 04:07:42.722612 5816 consensus_peers.cc:487] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 -> Peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Couldn't send request to peer df2f1e7811114b8a995f6771fb6a6c8a. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
W20260806 04:07:42.719976 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 71866.9us
I20260806 04:07:42.716465 7396 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:42.722779 4768 leader_election.cc:401] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260806 04:07:42.719563 4996 leader_election.cc:334] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.205s (SENT)
I20260806 04:07:42.724157 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:42.699506 7556 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:42.722980 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1892)
I20260806 04:07:42.710083 7588 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:42.734769 5816 consensus_peers.cc:487] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 -> Peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Couldn't send request to peer df2f1e7811114b8a995f6771fb6a6c8a. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:07:42.737136 4810 raft_consensus.cc:2354] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:07:42.709753 7607 raft_consensus.cc:2741] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 04:07:42.728935 4996 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:42.734225 4768 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:42.745057 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:42.747783 5037 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:42.748260 5030 raft_consensus.cc:1261] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:42.741196 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33548 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
I20260806 04:07:42.733057 4992 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:42.735250 7552 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 1.256s user 0.006s sys 0.006s
I20260806 04:07:42.721390 7378 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:42.694679 7497 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.753964 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53354 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1891)
I20260806 04:07:42.737488 7556 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:42.739727 7349 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:07:42.697700 7510 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.752058 7396 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:42.766923 4817 raft_consensus.cc:2354] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:42.756966 7511 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:42.765796 7479 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:42.741321 5856 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
W20260806 04:07:42.765978 7410 consensus_meta.cc:287] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.070s user 0.010s sys 0.016s
I20260806 04:07:43.059660 7410 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:42.753687 7407 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.060667 7407 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.062844 7425 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.062980 7407 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.768635 7497 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.063494 7425 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.697489 7283 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.065423 7283 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:43.065830 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1501) took 1008 ms. Trace:
I20260806 04:07:42.737839 7613 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:42.756641 7521 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:43.069003 7521 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:43.069434 7521 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:42.759960 7602 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:43.071449 7602 raft_consensus.cc:686] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:42.761940 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33548 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1887)
I20260806 04:07:42.830478 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 233) took 1529 ms. Trace:
W20260806 04:07:43.107613 4813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 230) took 1885 ms (client timeout 1552 ms). Trace:
W20260806 04:07:43.302218 4813 rpcz_store.cc:269] 0806 04:07:41.222610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.225615 (+ 3005us) service_pool.cc:227] Handling call
0806 04:07:43.107396 (+1881781us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b54000d07d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.107532 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1669390080}
W20260806 04:07:43.157186 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1186) took 1663 ms (client timeout 1510 ms). Trace:
W20260806 04:07:43.304544 4810 rpcz_store.cc:269] 0806 04:07:41.494128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.503729 (+ 9601us) service_pool.cc:227] Handling call
0806 04:07:43.156966 (+1653237us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540011ec54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.157122 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2834215680}
I20260806 04:07:42.918581 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1475) took 2326 ms. Trace:
I20260806 04:07:43.306123 4927 rpcz_store.cc:276] 0806 04:07:40.592626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.535907 (+943281us) service_pool.cc:227] Handling call
0806 04:07:41.536316 (+ 409us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.905739 (+369423us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.918260 (+1012521us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b540018f6d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:42.918493 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":680118144}
I20260806 04:07:43.302376 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33548 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213)
W20260806 04:07:42.785719 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 61612.1us
I20260806 04:07:42.900064 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1193) took 1229 ms. Trace:
I20260806 04:07:43.419649 4817 rpcz_store.cc:276] 0806 04:07:41.671079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.691942 (+ 20863us) service_pool.cc:227] Handling call
0806 04:07:42.899873 (+1207931us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003fd254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.900005 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2467683584}
I20260806 04:07:43.421403 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.129673 4816 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.423449 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:33548 (request call id 1886) took 2241 ms. Trace:
I20260806 04:07:42.859184 7612 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:43.422602 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1888) took 1046 ms. Trace:
I20260806 04:07:43.425314 7612 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:43.425697 4817 rpcz_store.cc:276] 0806 04:07:42.376112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.420871 (+1044759us) service_pool.cc:227] Handling call
0806 04:07:43.422470 (+ 1599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:42.750228 7616 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:43.297446 7621 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:43.427737 7621 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.058430 4882 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:43.590099 7566 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:07:43.073077 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 227) took 1851 ms (client timeout 2372 ms). Trace:
W20260806 04:07:42.905354 4919 consensus_meta.cc:287] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.580s user 0.013s sys 0.005s
I20260806 04:07:43.620288 4919 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:07:43.621196 4919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1499) took 1564 ms (client timeout 1900 ms). Trace:
W20260806 04:07:43.621630 4919 rpcz_store.cc:269] 0806 04:07:42.056844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.293037 (+236193us) service_pool.cc:227] Handling call
0806 04:07:43.621086 (+1328049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.066357 4920 rpcz_store.cc:276] 0806 04:07:42.057394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.478110 (+420716us) service_pool.cc:227] Handling call
0806 04:07:43.065537 (+587427us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54004137d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.065746 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":476671232}
I20260806 04:07:42.896628 4923 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:07:43.628731 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1498) took 1572 ms. Trace:
I20260806 04:07:43.629125 4923 rpcz_store.cc:276] 0806 04:07:42.056531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.698166 (+641635us) service_pool.cc:227] Handling call
0806 04:07:43.628504 (+930338us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:43.006482 4926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1492) took 2034 ms (client timeout 1763 ms). Trace:
W20260806 04:07:42.939808 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 206803us
I20260806 04:07:43.626374 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:43.300740 4996 leader_election.cc:334] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.903s (SENT)
I20260806 04:07:43.639533 4996 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:43.639279 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:43.642573 4919 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:43.116160 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1526) took 2280 ms. Trace:
W20260806 04:07:43.112227 5816 leader_election.cc:334] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.681s (SENT)
I20260806 04:07:43.314114 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:43.650755 4927 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:43.648005 5034 rpcz_store.cc:276] 0806 04:07:40.835880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.244995 (+409115us) service_pool.cc:227] Handling call
0806 04:07:41.245423 (+ 428us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:41.254109 (+ 8686us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.254158 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:41.254745 (+ 587us) log.cc:842] Serialized 26 byte log entry
0806 04:07:41.428451 (+173706us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:42.681041 (+1252590us) spinlock_profiling.cc:243] Waited 62.8 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:43.115403 (+434362us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b1800303f40. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a2f155 00007f23dd32f167 00007f23d9964807 00007f23d9973f0f
0806 04:07:43.115577 (+ 174us) log.cc:842] Serialized 14 byte log entry
0806 04:07:43.115820 (+ 243us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:43.115920 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:43.115958 (+ 38us) raft_consensus.cc:1671] finished
0806 04:07:43.115970 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:43.116098 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8182,"prepare.queue_time_us":331462,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":10257,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":19451,"spinlock_wait_cycles":153240576,"thread_start_us":407867,"threads_started":2,"wal-append.queue_time_us":215351}
I20260806 04:07:43.651258 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1506) took 1564 ms. Trace:
I20260806 04:07:43.651680 4927 rpcz_store.cc:276] 0806 04:07:42.087088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.308318 (+1221230us) service_pool.cc:227] Handling call
0806 04:07:43.651167 (+342849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.652983 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:42.882809 5853 raft_consensus.cc:2354] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:07:43.176815 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 502) took 1136 ms. Trace:
I20260806 04:07:42.941250 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:43.683439 5856 rpcz_store.cc:276] 0806 04:07:42.040424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.045726 (+ 5302us) service_pool.cc:227] Handling call
0806 04:07:42.117469 (+ 71743us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:43.064195 (+946726us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:43.176528 (+112333us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54005dded4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:43.176759 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1405244800}
I20260806 04:07:43.684859 5866 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:42.904964 5867 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 206) took 2148 ms (client timeout 2466 ms). Trace:
I20260806 04:07:43.686946 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:43.686877 5867 rpcz_store.cc:269] 0806 04:07:40.756582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:40.756720 (+ 138us) service_pool.cc:227] Handling call
0806 04:07:42.904751 (+2148031us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54004fced4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.904882 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3447473024}
I20260806 04:07:43.688521 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:42.907490 5854 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 527) took 3815 ms (client timeout 2190 ms). Trace:
I20260806 04:07:42.958556 5857 raft_consensus.cc:1226] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:43.111804 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 545) took 1932 ms. Trace:
I20260806 04:07:43.694099 5859 rpcz_store.cc:276] 0806 04:07:41.179286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.181913 (+ 2627us) service_pool.cc:227] Handling call
0806 04:07:41.187150 (+ 5237us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.759152 (+572002us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:43.111497 (+1352345us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54004448d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:43.111728 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1143609088}
I20260806 04:07:42.865067 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:43.128228 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 498) took 2022 ms. Trace:
I20260806 04:07:42.896486 5863 raft_consensus.cc:1226] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:43.698808 5862 rpcz_store.cc:276] 0806 04:07:41.105565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.106697 (+ 1132us) service_pool.cc:227] Handling call
0806 04:07:41.146670 (+ 39973us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.462090 (+315420us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.462119 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:41.462160 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:41.462197 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:43.127909 (+1665712us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54001a2054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:43.127968 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:43.128148 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724937600}
I20260806 04:07:42.899755 7099 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.955929 7182 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:43.705067 7182 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 04:07:42.827630 7184 consensus_meta.cc:287] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.684s user 0.007s sys 0.011s
I20260806 04:07:43.242470 7278 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:43.710605 7278 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.713050 7278 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.292593 7199 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.001s sys 0.010s
W20260806 04:07:43.716565 7199 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.340s user 0.005s sys 0.010s
W20260806 04:07:42.776558 7219 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.651s user 0.014s sys 0.004s
I20260806 04:07:43.718976 7219 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.720754 4818 raft_consensus.cc:2354] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:07:43.721398 7219 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.047302 7267 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:43.111125 7268 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:42.901105 7316 consensus_meta.cc:287] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.153s user 0.017s sys 0.000s
I20260806 04:07:42.948681 7326 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.726747 7316 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.863333 7371 raft_consensus.cc:3032] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:42.830049 7396 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:43.022714 7335 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.514s user 0.001s sys 0.010s
I20260806 04:07:43.052745 7419 consensus_queue.cc:260] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.732110 7396 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:42.831688 7193 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:43.735154 7193 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.735601 7396 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:42.903159 7461 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.113021 7345 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:43.030802 7390 consensus_meta.cc:287] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.545s user 0.007s sys 0.011s
I20260806 04:07:43.076002 7511 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.746618 7511 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:43.095243 7521 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.078527 7528 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.089176 7572 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.751971 7528 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:43.085873 7573 consensus_meta.cc:287] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.646s user 0.016s sys 0.005s
I20260806 04:07:43.754312 7528 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.754896 4927 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:43.754405 7573 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.756132 4931 raft_consensus.cc:2354] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:43.757017 7573 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.101094 7583 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:42.849082 7479 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.097738 7588 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:43.051580 7587 consensus_meta.cc:287] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.528s user 0.011s sys 0.005s
I20260806 04:07:43.762042 7479 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.761921 7587 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.764285 7587 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:42.907112 7510 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.089680 7515 consensus_meta.cc:287] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.978s user 0.005s sys 0.013s
I20260806 04:07:42.860743 7603 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.767259 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:43.109632 7543 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 1.614s user 0.003s sys 0.005s
I20260806 04:07:42.766709 7556 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:43.107697 7557 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.774499 7557 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:42.900753 7564 consensus_meta.cc:287] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.829s user 0.018s sys 0.001s
I20260806 04:07:42.947063 7567 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.777207 7564 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:43.145095 7584 log.cc:924] Time spent T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.117s user 0.008s sys 0.005s
W20260806 04:07:43.345340 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.008s sys 0.023s
W20260806 04:07:43.779872 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.022s user 0.008s sys 0.023s
I20260806 04:07:43.315881 4804 rpcz_store.cc:276] 0806 04:07:41.301015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.326684 (+ 25669us) service_pool.cc:227] Handling call
0806 04:07:41.409197 (+ 82513us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:41.409880 (+ 683us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.830233 (+1420353us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b5400021550. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.830414 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":188836736}
W20260806 04:07:43.780445 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 1.02589e+06us
I20260806 04:07:43.072352 7602 consensus_queue.cc:237] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.320302 4769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1213) took 1269 ms. Trace:
I20260806 04:07:43.782869 4769 rpcz_store.cc:276] 0806 04:07:42.051068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.320204 (+1269136us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 04:07:43.337962 7611 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.559s user 0.001s sys 0.006s
I20260806 04:07:43.392278 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:43.783556 4769 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33564 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898)
I20260806 04:07:43.415608 7618 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:07:43.786955 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 239) took 1403 ms (client timeout 1526 ms). Trace:
W20260806 04:07:43.787388 4809 rpcz_store.cc:269] 0806 04:07:42.383898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.391399 (+1007501us) service_pool.cc:227] Handling call
0806 04:07:43.786878 (+395479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.787027 7618 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.788501 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.788722 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1807) took 1272 ms. Trace:
I20260806 04:07:43.789203 4809 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:43.789806 7618 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:43.789935 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1894) took 1410 ms (client timeout 1607 ms). Trace:
I20260806 04:07:43.388612 6970 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.425884 7612 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:07:43.790465 4809 rpcz_store.cc:269] 0806 04:07:42.379711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.788082 (+1408371us) service_pool.cc:227] Handling call
0806 04:07:43.789882 (+ 1800us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.791157 6970 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:43.792342 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.106762 7613 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:43.434443 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.432461 7621 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.438724 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.794610 4817 raft_consensus.cc:2379] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:43.453291 7626 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.459297 7616 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:07:43.795408 4805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1210) took 1877 ms (client timeout 1903 ms). Trace:
I20260806 04:07:43.470793 7283 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:43.479907 7437 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.795820 7626 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.796291 7283 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.796526 7437 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.488668 7425 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.798379 7626 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.506508 7273 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:43.795423 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1206) took 1878 ms (client timeout 2058 ms). Trace:
W20260806 04:07:43.799763 4817 rpcz_store.cc:269] 0806 04:07:41.916880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.433942 (+1517062us) service_pool.cc:227] Handling call
0806 04:07:43.795336 (+361394us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.799594 7273 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.801045 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.512248 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:43.801978 7273 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.802562 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1895) took 1422 ms (client timeout 1582 ms). Trace:
W20260806 04:07:43.803031 4810 rpcz_store.cc:269] 0806 04:07:42.379987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.511835 (+1131848us) service_pool.cc:227] Handling call
0806 04:07:43.802491 (+290656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.789186 5036 rpcz_store.cc:276] 0806 04:07:42.515690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.515842 (+ 152us) service_pool.cc:227] Handling call
0806 04:07:42.521895 (+ 6053us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:42.522305 (+ 410us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:42.522336 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:42.522869 (+ 533us) log.cc:842] Serialized 26 byte log entry
0806 04:07:42.687042 (+164173us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:42.687090 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.687222 (+ 132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:43.336522 (+649300us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:43.788059 (+451537us) raft_consensus.cc:1671] finished
0806 04:07:43.788092 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:43.788457 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3121,"raft.queue_time_us":151393,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":10781,"spinlock_wait_cycles":38103936,"thread_start_us":163683,"threads_started":1,"wal-append.queue_time_us":227867}
I20260806 04:07:43.516768 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.519243 7280 consensus_queue.cc:1035] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:43.803871 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.804878 4810 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
W20260806 04:07:43.805269 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1199) took 1911 ms (client timeout 2106 ms). Trace:
I20260806 04:07:43.805446 7437 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.113577 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:07:43.805490 4810 rpcz_store.cc:269] 0806 04:07:41.894014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.803529 (+1909515us) service_pool.cc:227] Handling call
0806 04:07:43.805231 (+ 1702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.553208 4669 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46886 (request call id 287) took 8611 ms. Trace:
W20260806 04:07:43.591157 4814 rpcz_store.cc:269] 0806 04:07:41.221852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.243974 (+ 22122us) service_pool.cc:227] Handling call
0806 04:07:43.072787 (+1828813us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54003d8954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.072993 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":429204864}
I20260806 04:07:43.806989 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:43.807802 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:43.592728 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 534198us
W20260806 04:07:43.808748 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1197) took 2003 ms (client timeout 2200 ms). Trace:
W20260806 04:07:43.809193 4814 rpcz_store.cc:269] 0806 04:07:41.805290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.807424 (+2002134us) service_pool.cc:227] Handling call
0806 04:07:43.808678 (+ 1254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.591645 7566 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.630471 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:43.632136 4926 rpcz_store.cc:269] 0806 04:07:41.003169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.202508 (+199339us) service_pool.cc:227] Handling call
0806 04:07:43.006237 (+1803729us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54003543d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.006392 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":655436672}
I20260806 04:07:43.810993 7621 consensus_queue.cc:1046] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:43.811093 7566 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.812126 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:43.812353 7621 consensus_queue.cc:1046] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:43.813079 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1385) took 2100 ms. Trace:
I20260806 04:07:43.633975 7609 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:43.637776 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1507) took 1550 ms. Trace:
I20260806 04:07:43.813512 7587 raft_consensus.cc:3032] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:43.813643 7566 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.594317 4924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 244) took 2921 ms (client timeout 2937 ms). Trace:
I20260806 04:07:43.814105 7587 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:43.813995 4920 rpcz_store.cc:276] 0806 04:07:42.087462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.625877 (+1538415us) service_pool.cc:227] Handling call
0806 04:07:43.637644 (+ 11767us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.815447 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:43.174400 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1527) took 2338 ms. Trace:
I20260806 04:07:43.819839 5029 rpcz_store.cc:276] 0806 04:07:40.836246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.247566 (+411320us) service_pool.cc:227] Handling call
0806 04:07:42.325807 (+1078241us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:42.325999 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:42.326019 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:42.326054 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:42.326078 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:42.326129 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:43.174110 (+847981us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540037df50. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.174306 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2479033600}
W20260806 04:07:43.646462 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 345980us
I20260806 04:07:43.647162 7409 raft_consensus.cc:2741] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260806 04:07:43.795863 4805 rpcz_store.cc:269] 0806 04:07:41.918324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.438117 (+1519793us) service_pool.cc:227] Handling call
0806 04:07:43.795320 (+357203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.656903 4992 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:43.675432 4919 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
W20260806 04:07:43.801980 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1203) took 1886 ms (client timeout 2075 ms). Trace:
W20260806 04:07:43.682613 5853 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 554) took 1538 ms (client timeout 1859 ms). Trace:
I20260806 04:07:43.827040 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 262) took 1153 ms. Trace:
W20260806 04:07:43.690025 5854 rpcz_store.cc:269] 0806 04:07:39.092002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:39.138884 (+ 46882us) service_pool.cc:227] Handling call
0806 04:07:42.907269 (+3768385us) spinlock_profiling.cc:243] Waited 3.37 s on lock 0x7b54005eae54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:42.907402 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7743898368}
I20260806 04:07:43.695847 5856 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:43.827587 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:43.827538 4919 rpcz_store.cc:276] 0806 04:07:42.673570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.638729 (+965159us) service_pool.cc:227] Handling call
0806 04:07:43.826932 (+188203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.697309 5860 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:43.828601 4805 raft_consensus.cc:2362] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4. Re-sending same reply.
I20260806 04:07:43.828995 5860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 549) took 2293 ms. Trace:
W20260806 04:07:43.829193 4805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 234) took 1446 ms (client timeout 1681 ms). Trace:
I20260806 04:07:43.829298 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:43.709116 7184 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.703310 7408 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.831045 7408 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.832234 7184 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.833350 7408 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.714186 7182 raft_consensus.cc:3032] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:43.833966 7182 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
W20260806 04:07:43.723011 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1182) took 2229 ms (client timeout 2131 ms). Trace:
I20260806 04:07:43.729182 7316 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:43.839972 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 202391us
I20260806 04:07:43.841918 4992 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:43.842828 7616 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:43.843164 7613 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:07:43.827251 5853 rpcz_store.cc:269] 0806 04:07:42.144243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.147003 (+ 2760us) service_pool.cc:227] Handling call
0806 04:07:43.682229 (+1535226us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b5400445a54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.682436 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":872648960}
I20260806 04:07:43.843717 7613 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:43.844043 5812 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:43.732728 7335 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.844321 7613 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:43.024637 7308 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.739586 7461 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.739423 7003 consensus_queue.cc:1159] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7c76b6dcd8945cd818b8107caf4a174" responder_term: 2 status { last_received { term: 1 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:07:43.737461 7193 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.742488 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:43.750476 7521 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.757568 4931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1383) took 2045 ms (client timeout 1909 ms). Trace:
I20260806 04:07:43.760593 7583 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.849774 5866 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:43.857549 7566 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.834477 4919 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:43.864341 7308 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.864710 7566 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.865734 5867 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:43.865631 4931 rpcz_store.cc:269] 0806 04:07:41.712040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.104809 (+392769us) service_pool.cc:227] Handling call
0806 04:07:43.757287 (+1652478us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54000528d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.757473 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3704776448}
I20260806 04:07:43.859556 7280 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260806 04:07:43.867165 5863 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:43.846323 7335 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.826999 4817 rpcz_store.cc:269] 0806 04:07:41.915828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.800681 (+1884853us) service_pool.cc:227] Handling call
0806 04:07:43.801908 (+ 1227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.869091 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:43.869958 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1211) took 1951 ms (client timeout 2157 ms). Trace:
I20260806 04:07:43.870973 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:43.872542 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:43.872817 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:43.806816 4669 rpcz_store.cc:276] 0806 04:07:34.941296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:34.941558 (+ 262us) service_pool.cc:227] Handling call
0806 04:07:43.552967 (+8611409us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:39.089553 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:39.122000 (+ 32447us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:39.122076 (+ 76us) write_op.cc:435] Acquired schema lock
0806 04:07:39.122125 (+ 49us) tablet.cc:660] Decoding operations
0806 04:07:39.123077 (+ 952us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:39.123236 (+ 159us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:39.123300 (+ 64us) tablet.cc:683] Acquiring locks for 2 operations
0806 04:07:39.124570 (+ 1270us) tablet.cc:699] Row locks acquired
0806 04:07:39.124616 (+ 46us) write_op.cc:260] PREPARE: finished
0806 04:07:39.124901 (+ 285us) write_op.cc:270] Start()
0806 04:07:39.125054 (+ 153us) write_op.cc:276] Timestamp: P: 1785989259124847 usec, L: 0
0806 04:07:39.125083 (+ 29us) op_driver.cc:331] REPLICATION: starting
0806 04:07:39.253200 (+128117us) log.cc:842] Serialized 923 byte log entry
0806 04:07:40.605306 (+1352106us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b54001c1954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9956198 00007f23dd369ade 00007f23dd3692b8 00007f23dd36dba1 00007f23dd36db59 00007f23dd36dae9 00007f23dd36dab1 00007f23dd36cdad 00007f23dfc96204 00007f23dfc96039
0806 04:07:40.925126 (+319820us) op_driver.cc:444] REPLICATION: finished
0806 04:07:41.533113 (+607987us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d6a145fe 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23dd36a410
0806 04:07:42.060666 (+527553us) write_op.cc:301] APPLY: starting
0806 04:07:42.060795 (+ 129us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:42.060949 (+ 154us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:42.060981 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:42.142741 (+ 81760us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:42.168626 (+ 25885us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 04:07:42.234771 (+ 66145us) write_op.cc:312] APPLY: finished
0806 04:07:42.294787 (+ 60016us) log.cc:842] Serialized 29 byte log entry
0806 04:07:42.452703 (+157916us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:42.453743 (+ 1040us) write_op.cc:454] Released schema lock
0806 04:07:42.460372 (+ 6629us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:42.474666 (+ 14294us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:42.474722 (+ 56us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10341,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":1124375,"apply.run_cpu_time_us":20089,"apply.run_wall_time_us":414428,"num_ops":2,"prepare.queue_time_us":729843,"prepare.run_cpu_time_us":72941,"prepare.run_wall_time_us":1597217,"raft.queue_time_us":470865,"raft.run_cpu_time_us":156854,"raft.run_wall_time_us":1106649,"replication_time_us":1797056,"spinlock_wait_cycles":70814848,"thread_start_us":3339208,"threads_started":5,"wal-append.queue_time_us":940658,"wal-append.run_cpu_time_us":96358,"wal-append.run_wall_time_us":2820465}]]}
I20260806 04:07:43.875626 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:43.874157 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:43.764716 4927 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:42.733584 7527 consensus_queue.cc:1046] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260806 04:07:43.767144 7515 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:43.888787 7527 raft_consensus.cc:3032] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:43.889199 7527 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:43.889986 7515 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.769749 7603 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:43.771351 5857 raft_consensus.cc:2379] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:43.771698 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 219) took 2577 ms. Trace:
I20260806 04:07:43.775941 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:43.779824 7564 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:42.760787 7591 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:43.781474 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.781903 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1473) took 3615 ms. Trace:
I20260806 04:07:43.408133 7586 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:43.793330 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1200) took 1878 ms (client timeout 2075 ms). Trace:
I20260806 04:07:43.798379 7283 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.424363 4816 rpcz_store.cc:276] 0806 04:07:41.190985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.220221 (+ 29236us) service_pool.cc:227] Handling call
0806 04:07:43.128837 (+1908616us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54002bfcd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9951da7 00007f23de89dee0 00007f23d7f8f784 00007f23d7f8f712 00007f23d7f8f641 00007f23d7f8f5bc 00007f23d7f8e872 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.423185 (+294348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269903616}
I20260806 04:07:43.804129 4813 raft_consensus.cc:2354] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:43.524346 7570 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260806 04:07:43.807964 4810 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:43.815536 7587 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:43.814329 4924 rpcz_store.cc:269] 0806 04:07:40.672494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.458574 (+786080us) service_pool.cc:227] Handling call
0806 04:07:43.593829 (+2135255us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b54004132d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.594105 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2243775360}
I20260806 04:07:43.817634 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:43.645160 7484 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:43.811268 4923 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:43.873464 4919 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
W20260806 04:07:43.656172 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 543853us
I20260806 04:07:43.810344 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:43.829588 4805 rpcz_store.cc:269] 0806 04:07:42.382352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.824219 (+1441867us) service_pool.cc:227] Handling call
0806 04:07:43.829092 (+ 4873us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.833595 4928 raft_consensus.cc:2379] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:43.730425 7371 raft_consensus.cc:729] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:43.891283 4808 rpcz_store.cc:276] 0806 04:07:41.194199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.231280 (+ 37081us) service_pool.cc:227] Handling call
0806 04:07:41.231652 (+ 372us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:41.232072 (+ 420us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.232111 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:41.242584 (+ 10473us) log.cc:842] Serialized 26 byte log entry
0806 04:07:41.386177 (+143593us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:41.386222 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.386690 (+ 468us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:43.108492 (+1721802us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:43.771118 (+662626us) raft_consensus.cc:1671] finished
0806 04:07:43.771154 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:43.771463 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6771,"spinlock_wait_cycles":16037760,"thread_start_us":143293,"threads_started":1,"wal-append.queue_time_us":212519}
W20260806 04:07:43.912572 4804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1220) took 1664 ms (client timeout 1731 ms). Trace:
I20260806 04:07:43.829330 5860 rpcz_store.cc:276] 0806 04:07:41.535559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.535760 (+ 201us) service_pool.cc:227] Handling call
0806 04:07:42.864876 (+1329116us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:43.828891 (+964015us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6065920}
W20260806 04:07:43.913009 4804 rpcz_store.cc:269] 0806 04:07:42.250529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.780975 (+1530446us) service_pool.cc:227] Handling call
0806 04:07:43.912465 (+131490us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:43.835263 4818 rpcz_store.cc:269] 0806 04:07:41.493135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.130268 (+637133us) service_pool.cc:227] Handling call
0806 04:07:43.722636 (+1592368us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54003c1c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.722867 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3650030848}
I20260806 04:07:43.837975 7182 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.730406 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:43.870296 4817 rpcz_store.cc:269] 0806 04:07:41.918624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.868701 (+1950077us) service_pool.cc:227] Handling call
0806 04:07:43.869899 (+ 1198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.872845 4931 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:43.846251 4992 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:43.883611 5863 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:43.894461 7335 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:43.894918 5867 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:43.899842 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 259) took 1354 ms. Trace:
W20260806 04:07:43.902058 4809 rpcz_store.cc:269] 0806 04:07:41.914549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.791392 (+1876843us) service_pool.cc:227] Handling call
0806 04:07:43.793255 (+ 1863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.909948 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1500) took 1852 ms. Trace:
I20260806 04:07:43.813426 4926 rpcz_store.cc:276] 0806 04:07:41.712725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.811760 (+2099035us) service_pool.cc:227] Handling call
0806 04:07:43.812989 (+ 1229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.960646 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:43.969367 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1208) took 2009 ms (client timeout 2092 ms). Trace:
I20260806 04:07:43.969553 4927 rpcz_store.cc:276] 0806 04:07:42.545033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.652539 (+1107506us) service_pool.cc:227] Handling call
0806 04:07:43.899574 (+247035us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400355cd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.899750 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":232730880}
I20260806 04:07:43.871042 7566 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:43.911013 4813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1195) took 2156 ms (client timeout 2205 ms). Trace:
I20260806 04:07:43.918004 5862 raft_consensus.cc:2354] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
W20260806 04:07:43.920871 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 103458us
W20260806 04:07:43.969894 4814 rpcz_store.cc:269] 0806 04:07:41.917593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.809937 (+1892344us) service_pool.cc:227] Handling call
0806 04:07:43.927031 (+117094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.970834 4923 rpcz_store.cc:276] 0806 04:07:42.057111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.629952 (+1572841us) service_pool.cc:227] Handling call
0806 04:07:43.909886 (+279934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.896106 4930 rpcz_store.cc:276] 0806 04:07:40.166334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:41.504036 (+1337702us) service_pool.cc:227] Handling call
0806 04:07:41.504382 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:41.598344 (+ 93962us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:41.598394 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:41.636196 (+ 37802us) log.cc:842] Serialized 26 byte log entry
0806 04:07:41.806359 (+170163us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:41.806413 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:41.806492 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:43.144605 (+1338113us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:43.781201 (+636596us) raft_consensus.cc:1671] finished
0806 04:07:43.781243 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:43.781671 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56448,"raft.queue_time_us":33828,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":61409,"spinlock_wait_cycles":4216192,"thread_start_us":102265,"threads_started":1,"wal-append.queue_time_us":313267}
W20260806 04:07:43.972230 4813 rpcz_store.cc:269] 0806 04:07:41.754928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.516285 (+1761357us) service_pool.cc:227] Handling call
0806 04:07:43.910939 (+394654us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.923413 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:43.910384 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 87072us
I20260806 04:07:43.960129 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.974586 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.975112 4808 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:43.975356 4818 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:43.975660 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1196) took 2205 ms (client timeout 2320 ms). Trace:
W20260806 04:07:43.975876 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 236) took 1592 ms (client timeout 1729 ms). Trace:
W20260806 04:07:43.976020 4808 rpcz_store.cc:269] 0806 04:07:41.770479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.913142 (+2142663us) service_pool.cc:227] Handling call
0806 04:07:43.975578 (+ 62436us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:43.976215 4818 rpcz_store.cc:269] 0806 04:07:42.382982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.919003 (+1536021us) service_pool.cc:227] Handling call
0806 04:07:43.975828 (+ 56825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.977052 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.977171 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.977972 4808 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
W20260806 04:07:43.977970 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1893) took 1598 ms (client timeout 1803 ms). Trace:
W20260806 04:07:43.978541 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1223) took 1658 ms (client timeout 1845 ms). Trace:
W20260806 04:07:43.979027 4808 rpcz_store.cc:269] 0806 04:07:42.320090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.976743 (+1656653us) service_pool.cc:227] Handling call
0806 04:07:43.978494 (+ 1751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.980011 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:43.980783 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1900) took 1599 ms (client timeout 1826 ms). Trace:
W20260806 04:07:43.981123 4808 rpcz_store.cc:269] 0806 04:07:42.381460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.979703 (+1598243us) service_pool.cc:227] Handling call
0806 04:07:43.980742 (+ 1039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.982088 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.982815 7510 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.983052 4808 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
W20260806 04:07:43.983565 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1219) took 1749 ms (client timeout 1986 ms). Trace:
I20260806 04:07:43.983372 7510 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:43.983891 4808 rpcz_store.cc:269] 0806 04:07:42.234401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.981808 (+1747407us) service_pool.cc:227] Handling call
0806 04:07:43.983514 (+ 1706us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.984805 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.985462 4808 raft_consensus.cc:2354] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
W20260806 04:07:43.985824 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1215) took 1896 ms (client timeout 2131 ms). Trace:
I20260806 04:07:43.986021 7193 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:43.986173 4808 rpcz_store.cc:269] 0806 04:07:42.089893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.984537 (+1894644us) service_pool.cc:227] Handling call
0806 04:07:43.985781 (+ 1244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.986624 7193 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.987293 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.988077 4808 raft_consensus.cc:2354] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
W20260806 04:07:43.988597 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1224) took 1641 ms (client timeout 1884 ms). Trace:
W20260806 04:07:43.988890 4808 rpcz_store.cc:269] 0806 04:07:42.347475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.986859 (+1639384us) service_pool.cc:227] Handling call
0806 04:07:43.988553 (+ 1694us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.989519 7193 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:43.989830 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.985657 7510 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:43.978436 4818 rpcz_store.cc:269] 0806 04:07:42.379316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.976904 (+1597588us) service_pool.cc:227] Handling call
0806 04:07:43.977915 (+ 1011us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.992096 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.993182 4818 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:43.993346 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:43.993927 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1889) took 1617 ms (client timeout 1917 ms). Trace:
I20260806 04:07:43.993952 5863 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:43.961977 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:43.995541 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.995945 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:43.994459 4818 rpcz_store.cc:269] 0806 04:07:42.376474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.991615 (+1615141us) service_pool.cc:227] Handling call
0806 04:07:43.993828 (+ 2213us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:43.994429 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 238) took 1610 ms (client timeout 1897 ms). Trace:
W20260806 04:07:43.997110 4808 rpcz_store.cc:269] 0806 04:07:42.383580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.989530 (+1605950us) service_pool.cc:227] Handling call
0806 04:07:43.994330 (+ 4800us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:43.999069 4817 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:44.001220 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.910328 7308 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:44.001377 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1896) took 1621 ms (client timeout 1725 ms). Trace:
I20260806 04:07:44.004204 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.006121 4810 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:44.006495 4817 rpcz_store.cc:269] 0806 04:07:42.380285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.925438 (+1545153us) service_pool.cc:227] Handling call
0806 04:07:44.001323 (+ 75885us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.006871 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1212) took 2011 ms (client timeout 2007 ms). Trace:
W20260806 04:07:44.007469 4810 rpcz_store.cc:269] 0806 04:07:41.996780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.806603 (+1809823us) service_pool.cc:227] Handling call
0806 04:07:44.006791 (+200188us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.000195 4928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1384) took 2260 ms (client timeout 2320 ms). Trace:
I20260806 04:07:44.008502 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:44.010867 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:44.009168 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.011643 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:43.890190 7527 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:43.920001 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1504) took 1833 ms. Trace:
I20260806 04:07:43.973340 7587 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.008152 7182 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:44.013504 7521 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:44.013759 4927 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:44.014123 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.014323 7557 consensus_queue.cc:1046] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 04:07:44.015013 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 168901us
I20260806 04:07:44.015448 4770 leader_election.cc:398] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:07:44.016036 4813 raft_consensus.cc:2354] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:44.016611 4814 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:44.036715 4770 leader_election.cc:401] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260806 04:07:44.037045 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 250) took 1275 ms (client timeout 1552 ms). Trace:
I20260806 04:07:44.017877 4924 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:44.018847 7603 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:44.040192 7603 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.042357 7603 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:44.010811 7484 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:44.042860 4816 raft_consensus.cc:2354] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:44.008314 7335 raft_consensus.cc:686] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:07:44.043396 4816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1222) took 1739 ms (client timeout 1845 ms). Trace:
I20260806 04:07:44.043442 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:44.043885 4816 rpcz_store.cc:269] 0806 04:07:42.303647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.923006 (+1619359us) service_pool.cc:227] Handling call
0806 04:07:44.043339 (+120333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.044188 7335 consensus_queue.cc:237] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.044854 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.045017 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:44.045585 4816 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:44.046164 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1225) took 1435 ms. Trace:
I20260806 04:07:44.003386 4805 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:44.046468 4816 rpcz_store.cc:276] 0806 04:07:42.610464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.044561 (+1434097us) service_pool.cc:227] Handling call
0806 04:07:44.046111 (+ 1550us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.047061 4805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1897) took 1666 ms (client timeout 1709 ms). Trace:
I20260806 04:07:44.013536 4926 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:44.047984 7521 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:44.049254 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 243) took 3378 ms. Trace:
I20260806 04:07:44.049311 7521 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:44.051090 4882 leader_election.cc:334] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.593s (SENT)
I20260806 04:07:44.018503 7602 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:44.051852 7602 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:44.047508 4805 rpcz_store.cc:269] 0806 04:07:42.380598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.912093 (+1531495us) service_pool.cc:227] Handling call
0806 04:07:44.046983 (+134890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.053642 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.049829 4926 rpcz_store.cc:276] 0806 04:07:40.671228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.048069 (+3376841us) service_pool.cc:227] Handling call
0806 04:07:44.048445 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.048805 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:44.048843 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:44.048906 (+ 63us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:44.048947 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.049018 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:44.049154 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.054291 7591 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:44.055160 7609 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:44.056852 7609 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:44.057771 4992 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:44.021585 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.023311 5029 raft_consensus.cc:1226] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:44.023674 7616 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:44.021409 4768 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:44.019325 7316 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:43.923484 7586 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.016047 4995 leader_election.cc:398] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
W20260806 04:07:44.009938 4928 rpcz_store.cc:269] 0806 04:07:41.712402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.407988 (+695586us) service_pool.cc:227] Handling call
0806 04:07:43.919549 (+1511561us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540004ae54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:43.972253 (+ 52704us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3140847872}
I20260806 04:07:44.060534 4924 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:44.059216 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:44.034886 4804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1221) took 1760 ms (client timeout 1837 ms). Trace:
I20260806 04:07:44.037375 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.037451 4814 rpcz_store.cc:269] 0806 04:07:42.761412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.000869 (+1239457us) service_pool.cc:227] Handling call
0806 04:07:44.036983 (+ 36114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.038805 7557 raft_consensus.cc:3032] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:44.056283 7527 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.059643 4995 leader_election.cc:401] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:07:44.047480 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:43.908681 7612 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:44.060123 5863 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:44.062592 4818 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
W20260806 04:07:44.061306 4804 rpcz_store.cc:269] 0806 04:07:42.274460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.927653 (+1653193us) service_pool.cc:227] Handling call
0806 04:07:44.034622 (+106969us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.032902 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.059648 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1229) took 1337 ms (client timeout 1721 ms). Trace:
W20260806 04:07:44.063393 4813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1214) took 1992 ms (client timeout 2086 ms). Trace:
W20260806 04:07:44.063706 4809 rpcz_store.cc:269] 0806 04:07:42.722601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.002043 (+1279442us) service_pool.cc:227] Handling call
0806 04:07:44.059584 (+ 57541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.021243 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:44.063804 4813 rpcz_store.cc:269] 0806 04:07:42.070966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.998866 (+1927900us) service_pool.cc:227] Handling call
0806 04:07:44.063335 (+ 64469us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.056110 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 247) took 3370 ms. Trace:
I20260806 04:07:44.060070 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:44.067037 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 240) took 1585 ms (client timeout 2015 ms). Trace:
I20260806 04:07:44.067931 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1904) took 1327 ms. Trace:
W20260806 04:07:44.067667 4818 rpcz_store.cc:269] 0806 04:07:42.481425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.002013 (+1520588us) service_pool.cc:227] Handling call
0806 04:07:44.066977 (+ 64964us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.068300 4808 rpcz_store.cc:276] 0806 04:07:42.740713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.002589 (+1261876us) service_pool.cc:227] Handling call
0806 04:07:44.065891 (+ 63302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.065006 4926 rpcz_store.cc:276] 0806 04:07:40.685669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.054971 (+3369302us) service_pool.cc:227] Handling call
0806 04:07:44.055441 (+ 470us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.055739 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:44.055777 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:44.055815 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:44.055851 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.055910 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:44.056054 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.070366 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:44.069983 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.071051 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 49846.5us
I20260806 04:07:44.075943 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1481) took 3442 ms. Trace:
W20260806 04:07:44.076763 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 60973.5us
I20260806 04:07:44.069803 7316 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.077652 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.077940 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 65587.5us
I20260806 04:07:44.077885 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.054343 7602 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:44.078542 7586 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:43.913237 7371 consensus_queue.cc:260] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:44.079378 4816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1901) took 1687 ms (client timeout 2210 ms). Trace:
I20260806 04:07:44.078680 7510 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:44.123773 4805 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:44.337294 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:44.337847 7511 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:44.339522 4816 rpcz_store.cc:269] 0806 04:07:42.381771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.047166 (+1665395us) service_pool.cc:227] Handling call
0806 04:07:44.069175 (+ 22009us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.295527 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.020059 4919 rpcz_store.cc:276] 0806 04:07:42.086227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.872463 (+1786236us) service_pool.cc:227] Handling call
0806 04:07:43.917071 (+ 44608us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.340161 4924 rpcz_store.cc:276] 0806 04:07:40.625381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.046116 (+3420735us) service_pool.cc:227] Handling call
0806 04:07:44.046608 (+ 492us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.066917 (+ 20309us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.067160 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.341137 7371 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.341531 5036 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
W20260806 04:07:44.341854 4850 debug-util.cc:398] Leaking SignalData structure 0x7b08000e72a0 after lost signal to thread 3045
W20260806 04:07:44.343914 5816 leader_election.cc:334] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.897s (SENT)
I20260806 04:07:44.149480 7613 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:44.372035 7637 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260806 04:07:44.156955 7603 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.374071 7613 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:44.376260 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:44.064603 7557 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:44.377027 5859 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:44.346418 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.140638 4928 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:44.156509 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.347837 4810 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:44.071504 7484 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:44.359594 4814 raft_consensus.cc:2379] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:44.102352 4931 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:44.383723 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 263) took 1428 ms. Trace:
I20260806 04:07:44.384059 4931 rpcz_store.cc:276] 0806 04:07:42.955541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.870623 (+915082us) service_pool.cc:227] Handling call
0806 04:07:44.383662 (+513039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.353278 4927 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:44.384800 5029 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.185418 4674 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 04:07:44.376897 5816 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.015s (SENT)
I20260806 04:07:44.345069 7612 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:44.103638 7591 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
W20260806 04:07:44.390872 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 242) took 1908 ms (client timeout 2189 ms). Trace:
I20260806 04:07:44.201804 7594 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:44.352388 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 294718us
I20260806 04:07:44.354278 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:44.386987 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 243) took 1904 ms (client timeout 2100 ms). Trace:
I20260806 04:07:44.224050 7609 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:44.389156 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.390021 4930 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
W20260806 04:07:44.391270 4814 rpcz_store.cc:269] 0806 04:07:42.481989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.061657 (+1579668us) service_pool.cc:227] Handling call
0806 04:07:44.390808 (+329151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.388294 4674 catalog_manager.cc:5495] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 4 to 6, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 6 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.396168 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 249) took 3236 ms. Trace:
W20260806 04:07:44.396826 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1191) took 2727 ms (client timeout 3080 ms). Trace:
W20260806 04:07:44.400431 4809 rpcz_store.cc:269] 0806 04:07:41.669614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.068886 (+2399272us) service_pool.cc:227] Handling call
0806 04:07:44.396753 (+327867us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.399346 4810 rpcz_store.cc:269] 0806 04:07:42.482300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.010012 (+1527712us) service_pool.cc:227] Handling call
0806 04:07:44.386792 (+376780us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54003d8454. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:44.386914 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":231201536}
I20260806 04:07:44.401973 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.402881 4810 raft_consensus.cc:2354] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:44.403378 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1230) took 1103 ms. Trace:
I20260806 04:07:44.387158 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.400341 4928 rpcz_store.cc:276] 0806 04:07:41.151235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.084865 (+2933630us) service_pool.cc:227] Handling call
0806 04:07:44.140419 (+ 55554us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.383014 (+242595us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.388070 (+ 5056us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.403656 4674 catalog_manager.cc:5495] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.403766 4810 rpcz_store.cc:276] 0806 04:07:43.299787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.401624 (+1101837us) service_pool.cc:227] Handling call
0806 04:07:44.403311 (+ 1687us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.406315 4818 raft_consensus.cc:2379] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:44.406828 4768 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:44.370604 4882 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.725s (SENT) [suppressed 67 similar messages]
I20260806 04:07:44.409970 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 248) took 3258 ms. Trace:
W20260806 04:07:44.408529 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1903) took 1668 ms (client timeout 2032 ms). Trace:
I20260806 04:07:44.411768 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.411015 4818 rpcz_store.cc:269] 0806 04:07:42.740331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.069539 (+1329208us) service_pool.cc:227] Handling call
0806 04:07:44.347369 (+277830us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:44.408331 (+ 60962us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18040064}
I20260806 04:07:44.382123 7510 raft_consensus.cc:686] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:44.431658 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 251) took 2957 ms. Trace:
I20260806 04:07:44.080989 7616 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:44.163408 7556 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.411299 7511 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:44.451331 7419 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:44.455329 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.563766 5862 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:44.564177 4809 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:44.564745 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 246) took 1875 ms (client timeout 2233 ms). Trace:
W20260806 04:07:44.564976 4809 rpcz_store.cc:269] 0806 04:07:42.689630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.401273 (+1711643us) service_pool.cc:227] Handling call
0806 04:07:44.564701 (+163428us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.456329 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1227) took 1778 ms (client timeout 2147 ms). Trace:
W20260806 04:07:44.474498 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 103886us
W20260806 04:07:44.466789 4850 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4360 after lost signal to thread 4724
I20260806 04:07:44.495352 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.379690 7557 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.515038 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.552273 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 141108us
I20260806 04:07:44.410362 4930 rpcz_store.cc:276] 0806 04:07:41.151059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.081272 (+2930213us) service_pool.cc:227] Handling call
0806 04:07:44.366204 (+284932us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.409311 (+ 43107us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.409910 (+ 599us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.557435 4816 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:44.408211 4804 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:44.558166 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1231) took 1248 ms. Trace:
I20260806 04:07:44.558429 4924 rpcz_store.cc:276] 0806 04:07:41.451386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.368797 (+2917411us) service_pool.cc:227] Handling call
0806 04:07:44.387076 (+ 18279us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.396843 (+ 9767us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:44.396883 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:44.408734 (+ 11851us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:44.408788 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.408843 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:44.408954 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.571096 4816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1228) took 1893 ms (client timeout 2157 ms). Trace:
I20260806 04:07:44.559274 4674 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.560482 5034 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:44.561287 4805 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:44.561515 5029 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:44.561774 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:44.565909 4808 rpcz_store.cc:269] 0806 04:07:42.677835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.348679 (+1670844us) service_pool.cc:227] Handling call
0806 04:07:44.456222 (+107543us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.434325 7594 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:44.567461 4926 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 6. Ops: [4.7-4.7]
I20260806 04:07:44.568854 7587 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.578115 7511 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:44.578811 4882 leader_election.cc:334] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.803s (SENT)
I20260806 04:07:44.579066 5859 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:44.579327 4882 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:44.579649 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.580307 4808 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
W20260806 04:07:44.580674 4882 leader_election.cc:334] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.826s (SENT)
W20260806 04:07:44.572014 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1198) took 2698 ms (client timeout 3029 ms). Trace:
W20260806 04:07:44.581862 4814 rpcz_store.cc:269] 0806 04:07:41.873977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.400026 (+2526049us) service_pool.cc:227] Handling call
0806 04:07:44.571955 (+171929us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.567267 7527 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.500s user 0.011s sys 0.006s
I20260806 04:07:44.571208 4810 rpcz_store.cc:276] 0806 04:07:43.309629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.406166 (+1096537us) service_pool.cc:227] Handling call
0806 04:07:44.558091 (+151925us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.573320 4927 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
W20260806 04:07:44.571146 4804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1226) took 1960 ms (client timeout 2097 ms). Trace:
I20260806 04:07:44.575007 4920 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:44.576210 4931 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:44.576591 5036 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:44.584571 4804 rpcz_store.cc:269] 0806 04:07:42.610864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.063534 (+1452670us) service_pool.cc:227] Handling call
0806 04:07:44.571095 (+507561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.585210 5036 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:44.576975 4924 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:44.569953 4850 debug-util.cc:398] Leaking SignalData structure 0x7b08000e42a0 after lost signal to thread 4744
I20260806 04:07:44.586160 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 258) took 2660 ms. Trace:
I20260806 04:07:44.586831 4924 rpcz_store.cc:276] 0806 04:07:41.925916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.574326 (+2648410us) service_pool.cc:227] Handling call
0806 04:07:44.576767 (+ 2441us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.585873 (+ 9106us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.586066 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.572022 4816 rpcz_store.cc:269] 0806 04:07:42.678181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.369252 (+1691071us) service_pool.cc:227] Handling call
0806 04:07:44.571036 (+201784us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.583856 4674 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 3, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.581398 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.588413 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1395) took 2866 ms. Trace:
I20260806 04:07:44.588613 4808 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:44.587895 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 250) took 3430 ms. Trace:
I20260806 04:07:44.589118 4926 rpcz_store.cc:276] 0806 04:07:41.151519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.380294 (+3228775us) service_pool.cc:227] Handling call
0806 04:07:44.560920 (+180626us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:44.582080 (+ 21160us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.582292 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.588670 4928 rpcz_store.cc:276] 0806 04:07:41.715969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.412041 (+2696072us) service_pool.cc:227] Handling call
0806 04:07:44.572530 (+160489us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.582162 (+ 9632us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:44.582198 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:44.582240 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:44.582271 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.582328 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:44.582485 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.591271 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1390) took 2877 ms. Trace:
I20260806 04:07:44.591579 4931 rpcz_store.cc:276] 0806 04:07:41.714308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.411204 (+2696896us) service_pool.cc:227] Handling call
0806 04:07:44.573980 (+162776us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.591062 (+ 17082us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.591224 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.592623 4923 raft_consensus.cc:2396] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:07:44.586059 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1393) took 2869 ms. Trace:
I20260806 04:07:44.592621 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.593305 4927 rpcz_store.cc:276] 0806 04:07:41.715340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.429763 (+2714423us) service_pool.cc:227] Handling call
0806 04:07:44.573152 (+143389us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.584175 (+ 11023us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.584357 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:44.592584 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 221674us
W20260806 04:07:44.593639 4805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 237) took 2204 ms (client timeout 2265 ms). Trace:
W20260806 04:07:44.594564 4805 rpcz_store.cc:269] 0806 04:07:42.383265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.053202 (+1669937us) service_pool.cc:227] Handling call
0806 04:07:44.585960 (+532758us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54000afad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:44.587830 (+ 1870us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":159645056}
I20260806 04:07:44.595853 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.596505 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.597015 4809 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:44.597299 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1898) took 2216 ms. Trace:
I20260806 04:07:44.597467 4809 rpcz_store.cc:276] 0806 04:07:42.380893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.565632 (+2184739us) service_pool.cc:227] Handling call
0806 04:07:44.597263 (+ 31631us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.599584 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:44.600147 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 223305us
I20260806 04:07:44.601203 4930 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 4. Ops: []
I20260806 04:07:44.599196 7367 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:44.601854 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1398) took 2885 ms. Trace:
I20260806 04:07:44.602214 4930 rpcz_store.cc:276] 0806 04:07:41.716865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.570253 (+2853388us) service_pool.cc:227] Handling call
0806 04:07:44.600990 (+ 30737us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.601598 (+ 608us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.601798 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:44.586407 7591 raft_consensus.cc:686] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:44.603683 7591 consensus_queue.cc:237] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:44.586155 7609 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.607043 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.607761 4814 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 1.
I20260806 04:07:44.608671 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.600536 4816 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:44.609230 4814 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:44.596405 4674 catalog_manager.cc:5495] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.616629 4674 catalog_manager.cc:5495] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.621212 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.616109 4992 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:44.621927 4808 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:44.602720 4850 debug-util.cc:398] Leaking SignalData structure 0x7b080019b080 after lost signal to thread 4853
I20260806 04:07:44.620800 4674 catalog_manager.cc:5495] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:44.632186 4674 catalog_manager.cc:5495] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:44.632433 7510 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.634238 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:44.633936 4674 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 4, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 4 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:44.635139 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.635416 7557 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:44.635866 4814 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.636193 4768 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:44.637552 7521 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:44.638221 7521 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:44.634965 7511 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:44.639304 4992 leader_election.cc:398] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:44.639750 4992 leader_election.cc:401] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:07:44.639751 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.640743 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:44.638568 7521 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:44.643395 7521 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.643347 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:44.645103 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.645355 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.642577 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.644358 7594 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:44.638830 4850 debug-util.cc:398] Leaking SignalData structure 0x7b0800369e60 after lost signal to thread 4856
I20260806 04:07:44.620748 7616 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:44.650902 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.647913 4809 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:44.641903 5816 leader_election.cc:334] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.100s (SENT)
I20260806 04:07:44.641815 7511 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:44.653635 4816 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:44.653769 7511 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.654066 7335 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:44.654577 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.655421 4816 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:44.656095 5812 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:44.655656 7556 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:44.642921 7316 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:44.710749 4817 raft_consensus.cc:2396] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:07:44.658000 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.663674 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1505) took 2569 ms. Trace:
I20260806 04:07:44.711414 4924 rpcz_store.cc:276] 0806 04:07:42.086576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.594849 (+2508273us) service_pool.cc:227] Handling call
0806 04:07:44.648237 (+ 53388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:44.655738 (+ 7501us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:44.655774 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:44.655818 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:44.655852 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.655912 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:44.656157 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1238,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186}
I20260806 04:07:44.695533 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.712809 4805 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 7
W20260806 04:07:44.702628 7640 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 04:07:44.703143 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.708526 5034 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:44.708993 5816 leader_election.cc:334] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.189s (SENT)
I20260806 04:07:44.733376 5816 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:44.733906 4808 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:44.735327 4925 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:44.739287 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.740038 4809 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:44.740057 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.736487 4814 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:44.659605 4850 debug-util.cc:398] Leaking SignalData structure 0x7b080036cb80 after lost signal to thread 4966
I20260806 04:07:44.744740 4814 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:44.745836 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.746949 4810 raft_consensus.cc:2354] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:44.741899 4816 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
W20260806 04:07:44.745301 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 730174us
I20260806 04:07:44.732949 7616 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:44.782233 4816 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 04:07:44.787289 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 152776us
I20260806 04:07:44.790414 4808 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:44.785104 4816 raft_consensus.cc:1226] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:44.750073 7030 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:44.794811 4770 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:44.800231 4770 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:44.750923 7603 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:07:44.805616 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.810920 4818 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:44.816474 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:44.819053 5026 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:44.750475 7371 consensus_queue.cc:1046] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
W20260806 04:07:44.832160 4882 leader_election.cc:334] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.210s (SENT)
I20260806 04:07:44.834378 4882 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:44.808730 7586 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:44.833308 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 177288us
I20260806 04:07:44.834436 5034 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:44.822543 4850 debug-util.cc:398] Leaking SignalData structure 0x7b0800356dc0 after lost signal to thread 5070
I20260806 04:07:44.825356 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.844449 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:44.828411 7557 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:44.856504 4805 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
W20260806 04:07:44.863821 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 63694.4us
I20260806 04:07:44.865561 5030 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.818024 7030 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.867357 7653 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:44.867806 7653 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:44.851889 7335 raft_consensus.cc:686] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:44.868193 7653 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:44.868350 7655 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:44.869046 7335 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.869772 7655 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:44.838447 7586 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:44.864730 7371 consensus_queue.cc:1225] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer 22fe557f852d4415a96fd683f970cc24 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:07:44.878419 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.879277 5032 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:44.880404 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.880746 7510 consensus_queue.cc:1046] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:44.881203 5026 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:44.880927 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.852094 7419 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:44.880429 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.883847 5032 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:44.882002 5028 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:44.879272 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.885185 5035 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:44.885864 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:07:44.949918 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1217) took 2795 ms (client timeout 2424 ms). Trace:
W20260806 04:07:44.943069 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 117093us
I20260806 04:07:44.895752 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:44.918777 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:44.886932 4925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 257) took 3199 ms (client timeout 2265 ms). Trace:
I20260806 04:07:44.961412 5034 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:44.936443 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:44.962772 4924 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.960506 5037 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:44.886102 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:44.961755 4925 rpcz_store.cc:269] 0806 04:07:41.687322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:42.285594 (+598272us) service_pool.cc:227] Handling call
0806 04:07:44.886677 (+2601083us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54003548d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:44.886837 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5563143936}
I20260806 04:07:44.966202 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:44.888773 6970 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:44.966912 4804 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.970072 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:45392 (request call id 435) took 3271 ms. Trace:
I20260806 04:07:44.971449 5032 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:44.961061 5028 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:44.930862 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 221915us
I20260806 04:07:44.865006 7557 raft_consensus.cc:686] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:44.970521 4928 rpcz_store.cc:276] 0806 04:07:41.698223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.614925 (+2916702us) service_pool.cc:227] Handling call
0806 04:07:44.969448 (+354523us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b54003548d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23de89fc6e 00007f23d7f96807 00007f23d7f96792 00007f23d7f966c1 00007f23d7f9663c 00007f23d7f958f2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:44.969965 (+ 517us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77501696}
W20260806 04:07:44.960561 4817 rpcz_store.cc:269] 0806 04:07:42.154769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.007583 (+1852814us) service_pool.cc:227] Handling call
0806 04:07:44.947809 (+940226us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b540011dd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:44.949815 (+ 2006us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1487675136}
I20260806 04:07:44.882445 7609 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:44.882270 7586 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:44.983225 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:44.917030 7655 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:44.889547 7652 raft_consensus.cc:3032] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:44.998876 7652 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:44.999353 4810 raft_consensus.cc:3032] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:44.999755 4810 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:44.999933 7652 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.000718 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.000583 4810 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.001574 4930 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:44.982293 5035 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:45.001860 4810 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:44.979943 4850 debug-util.cc:398] Leaking SignalData structure 0x7b0800112340 after lost signal to thread 5901
I20260806 04:07:44.995530 5032 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:44.987501 6970 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.995965 7655 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:44.996932 7511 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:44.997788 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:44.975237 4919 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:45.001888 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.019668 5026 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:45.007400 5028 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:45.006798 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.021570 5034 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:45.012565 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1236) took 1159 ms. Trace:
W20260806 04:07:45.015611 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 147875us
W20260806 04:07:45.022591 4813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1899) took 2641 ms (client timeout 2371 ms). Trace:
I20260806 04:07:45.022616 4818 rpcz_store.cc:276] 0806 04:07:43.853091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.468166 (+615075us) service_pool.cc:227] Handling call
0806 04:07:45.012286 (+544120us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540040f954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:45.012476 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":554400000}
I20260806 04:07:45.017915 4925 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4. Ops: []
W20260806 04:07:45.023074 4813 rpcz_store.cc:269] 0806 04:07:42.381177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.069068 (+1687891us) service_pool.cc:227] Handling call
0806 04:07:45.014158 (+945090us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b54000230d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:45.022505 (+ 8347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1213652736}
W20260806 04:07:45.038419 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 45575.9us
I20260806 04:07:44.888854 7316 consensus_queue.cc:1046] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.131s
I20260806 04:07:45.037587 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.040807 4927 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.035702 7511 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:45.046554 4920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 260) took 2409 ms (client timeout 2233 ms). Trace:
I20260806 04:07:45.051002 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:45.050930 4920 rpcz_store.cc:269] 0806 04:07:42.637559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.815022 (+1177463us) service_pool.cc:227] Handling call
0806 04:07:45.046293 (+1231271us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b540006de54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:45.046463 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1744493952}
W20260806 04:07:45.061101 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 88280.9us
I20260806 04:07:45.061597 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.065716 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:45.066615 7556 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:45.066771 7655 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.067422 7556 consensus_queue.cc:1046] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:45.067711 4876 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:45.068914 5816 leader_election.cc:334] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.552s (SENT)
I20260806 04:07:45.069365 6970 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.069497 7594 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:45.069913 7594 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:45.070230 7594 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.070505 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:45.071343 5862 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:45.074000 5034 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:45.076011 5026 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:45.080313 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.080933 4928 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:45.051265 7609 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.083118 7511 consensus_queue.cc:1046] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:45.083614 7609 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.047922 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 252) took 3590 ms. Trace:
I20260806 04:07:45.083995 7511 consensus_queue.cc:1225] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer d7c76b6dcd8945cd818b8107caf4a174 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 04:07:45.085148 4930 raft_consensus.cc:3032] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:45.090008 7030 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.090898 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 265) took 1480 ms. Trace:
I20260806 04:07:45.097393 4930 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:45.097544 4923 rpcz_store.cc:276] 0806 04:07:43.610702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.998919 (+388217us) service_pool.cc:227] Handling call
0806 04:07:45.090634 (+1091715us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54004132d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:45.090836 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332180864}
I20260806 04:07:45.084015 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:45.094060 4804 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:45.101058 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0394689231179655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
I20260806 04:07:45.058954 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:45.101210 7652 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.101671 7371 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:45.112327 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0331689231097698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20260806 04:07:45.076129 4996 connection.cc:232] RPC call timeout handler was delayed by -1.1764866018202156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
I20260806 04:07:45.107950 7603 raft_consensus.cc:3032] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
W20260806 04:07:45.113399 4996 connection.cc:232] RPC call timeout handler was delayed by -1.1575278989039361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
I20260806 04:07:45.092145 4919 rpcz_store.cc:276] 0806 04:07:41.452364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.391793 (+2939429us) service_pool.cc:227] Handling call
0806 04:07:44.966186 (+574393us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b54001b8428. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23de8017f1 00007f23de87c2dc 00007f23de8b037c 00007f23de89b80b 00007f23de89b2a2 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c
0806 04:07:44.966336 (+ 150us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:45.019187 (+ 52851us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:45.040208 (+ 21021us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17471360}
I20260806 04:07:45.114892 5037 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:45.115021 7371 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.115579 4930 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.116842 4930 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.117924 7371 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.120561 4926 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260806 04:07:45.127336 4924 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.121044 4926 pending_rounds.cc:76] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Aborting all ops after (but not including) 4
W20260806 04:07:45.131520 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 63907.6us
I20260806 04:07:45.136263 4876 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.138424 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:45.138928 4996 leader_election.cc:334] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.731s (SENT)
I20260806 04:07:45.139770 5030 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:45.139135 5863 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:45.141561 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:45.135931 4926 pending_rounds.cc:97] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 04:07:45.142186 4926 raft_consensus.cc:2866] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:45.143884 7527 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:45.120225 7603 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:45.149291 4810 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:45.149499 4876 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:45.151927 4996 leader_election.cc:334] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.837s (SENT)
I20260806 04:07:45.150300 7527 raft_consensus.cc:686] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:45.081638 7557 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.155517 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:45.153801 7643 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260806 04:07:45.164176 4924 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:45.165185 4923 raft_consensus.cc:1261] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:45.163168 7419 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 04:07:45.155046 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:45.171041 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.171700 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.172171 7316 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:45.172467 4808 raft_consensus.cc:1226] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:45.188527 4925 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.172633 7335 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:45.173429 7616 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:45.174082 7603 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.162880 7030 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:45.198150 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.199008 4925 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:45.177479 7655 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:45.173429 7521 consensus_meta.cc:287] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.523s user 0.008s sys 0.003s
I20260806 04:07:45.181984 7653 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.182614 7527 consensus_queue.cc:237] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.210801 7521 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.212291 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:45.212980 7521 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.212801 7586 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.213593 7521 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:45.214923 6970 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:45.216539 7603 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:45.216671 7521 raft_consensus.cc:686] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:45.195379 5859 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.220477 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.220781 7316 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.221323 4808 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:45.221562 7316 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.198427 7030 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:45.223037 7030 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:45.224280 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.224584 5862 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:45.224902 5038 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.226013 5863 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:45.206847 4920 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:45.207803 7653 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:45.191339 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.217677 7669 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:45.231655 7587 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.654s user 0.007s sys 0.005s
I20260806 04:07:45.232429 7587 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:07:45.232745 7609 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.236421 4808 raft_consensus.cc:1226] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:45.236861 4810 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:45.237310 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.238116 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.227597 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.239487 5030 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.240024 4818 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:45.240590 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.201007 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 261) took 2558 ms. Trace:
I20260806 04:07:45.241150 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.241539 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.241460 4926 rpcz_store.cc:276] 0806 04:07:42.642335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.610255 (+1967920us) service_pool.cc:227] Handling call
0806 04:07:45.091330 (+481075us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:45.148733 (+ 57403us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:45.200699 (+ 51966us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b54004132d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:45.200932 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1015592192}
I20260806 04:07:45.227794 7616 consensus_queue.cc:1046] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260806 04:07:45.243244 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.243842 7616 raft_consensus.cc:3032] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:45.243892 4818 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:45.244302 7616 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:45.228740 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.245241 7616 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.228917 7669 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.248792 7669 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.238152 4813 raft_consensus.cc:2379] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:45.229795 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.250461 7616 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.229632 7653 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260806 04:07:45.250792 5029 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.228389 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.249312 7594 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.228662 4927 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.252063 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.254967 4810 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:45.242508 4816 raft_consensus.cc:2396] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 04:07:45.256683 5812 leader_election.cc:334] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.526s (SENT)
I20260806 04:07:45.241175 4817 raft_consensus.cc:2362] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:45.257752 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.229162 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:45.248689 4930 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.258845 5863 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.227313 7586 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.227231 7521 consensus_queue.cc:237] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.261301 5862 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:45.265341 7556 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260806 04:07:45.265794 5859 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:45.266968 5812 leader_election.cc:334] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.592s (SENT)
I20260806 04:07:45.229394 7511 consensus_queue.cc:1046] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260806 04:07:45.269060 4805 raft_consensus.cc:2362] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3. Re-sending same reply.
I20260806 04:07:45.269948 5863 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.254299 7594 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.273943 4927 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.288292 5862 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:07:45.303601 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.304697 4927 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:45.304967 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 146035us
I20260806 04:07:45.307097 5036 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:45.308929 4876 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.310376 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:45.313488 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.319540 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.320353 4805 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.320988 4876 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.324455 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.325397 4926 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 04:07:45.325791 4926 pending_rounds.cc:76] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Aborting all ops after (but not including) 1
I20260806 04:07:45.326116 4926 pending_rounds.cc:97] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 04:07:45.326373 4926 raft_consensus.cc:2866] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:45.325416 5026 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:45.326968 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.327754 4924 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.324507 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.328563 4808 raft_consensus.cc:2379] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:45.317741 7587 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.326390 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.329084 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.327530 4876 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:45.340298 5812 leader_election.cc:334] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.153s (SENT)
I20260806 04:07:45.333128 4928 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:07:45.329634 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:45.339568 5029 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:45.339893 4817 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:45.338478 4805 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:45.329493 7556 raft_consensus.cc:2741] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 04:07:45.345098 4927 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:45.340457 4876 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:45.333638 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 181738us
W20260806 04:07:45.346626 4996 leader_election.cc:334] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.845s (SENT)
W20260806 04:07:45.347872 4996 leader_election.cc:334] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.845s (SENT)
I20260806 04:07:45.340958 5812 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.323288 7594 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:07:45.348850 4996 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.007s (SENT)
I20260806 04:07:45.350561 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.326515 7669 raft_consensus.cc:2741] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:07:45.376327 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.350209 5038 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:45.375252 4996 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.378180 4876 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:45.381865 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.383229 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:45.383462 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 107698us
I20260806 04:07:45.342005 7674 raft_consensus.cc:2741] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 04:07:45.383981 4808 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:45.384670 5026 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:45.385538 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.385715 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.387102 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.326733 7510 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260806 04:07:45.378374 7556 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:45.393119 7556 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:45.376964 7609 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:45.393553 7556 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.394114 7609 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:45.338806 7557 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
W20260806 04:07:45.399586 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 72857.8us
I20260806 04:07:45.384506 7653 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:45.411121 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.411960 4814 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.412845 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.415304 5026 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:45.416492 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.417183 5026 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.417958 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.387398 7671 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:45.418870 7671 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:45.419209 7671 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.420365 5029 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:07:45.422888 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.423053 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.424304 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.425098 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.420362 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:45.423243 7316 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.406672 5812 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:45.429318 4814 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.435266 5026 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.349131 7587 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:45.439483 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 90636.3us
I20260806 04:07:45.424899 4816 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:45.444016 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.398039 7674 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:07:45.449841 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.450573 4814 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.447425 5029 raft_consensus.cc:1261] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:45.453855 5032 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:45.401952 7511 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:45.449872 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.457854 5038 raft_consensus.cc:2379] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:45.395155 7609 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.466913 4813 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 04:07:45.424469 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:45.472013 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.472718 5038 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:45.473394 4996 leader_election.cc:334] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.086s (SENT)
I20260806 04:07:45.475533 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.449009 7655 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:45.479614 4996 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:45.477869 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 92230.2us
I20260806 04:07:45.478120 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.482461 4804 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:45.488495 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:45.495285 5038 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:07:45.495849 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:45.491556 4816 raft_consensus.cc:2362] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
I20260806 04:07:45.494791 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.494735 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.467479 7674 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:45.491416 5816 leader_election.cc:334] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.372s (SENT)
I20260806 04:07:45.493479 4814 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:45.491263 7527 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.501122 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.510537 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.508787 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.518596 4810 raft_consensus.cc:2354] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3.
I20260806 04:07:45.512941 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.520256 4809 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:45.502327 5035 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.516000 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.522212 5038 raft_consensus.cc:2379] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.503628 4876 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.513878 4813 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
W20260806 04:07:45.534272 5812 leader_election.cc:334] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.889s (SENT)
I20260806 04:07:45.495047 7586 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.501724 7669 raft_consensus.cc:2741] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 04:07:45.501969 7678 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:45.544632 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.552331 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:45.578483 4813 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.578684 4876 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.554418 5030 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.556079 7661 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:45.503149 7655 raft_consensus.cc:686] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:45.560272 7419 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:07:45.509829 7668 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.354s user 0.002s sys 0.000s
W20260806 04:07:45.508759 7639 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.946s user 0.010s sys 0.010s
I20260806 04:07:45.571986 5026 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.512282 7603 raft_consensus.cc:3032] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 04:07:45.584627 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 119357us
I20260806 04:07:45.609066 7691 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:45.608508 7673 log.cc:924] Time spent T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20260806 04:07:45.657938 7677 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.268s user 0.002s sys 0.000s
I20260806 04:07:45.590205 7419 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:45.659011 7419 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.658849 6970 consensus_queue.cc:1046] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 04:07:45.600015 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 65745.2us
I20260806 04:07:45.647205 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.507061 7527 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.667745 7527 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.668308 7527 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:45.668645 7527 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:45.668978 7527 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.590377 7030 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.672662 7030 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.673259 7030 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:45.673485 5026 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.673630 7030 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:45.674052 7030 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:45.611093 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.676517 4924 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:45.676674 5034 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:45.681870 4996 leader_election.cc:334] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.131s (SENT)
I20260806 04:07:45.570552 7586 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.689476 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.628497 7594 raft_consensus.cc:686] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:45.699645 5026 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.647455 7603 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:45.705727 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.716917 5026 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:07:45.718713 5853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 230) took 1844 ms. Trace:
I20260806 04:07:45.719214 5853 rpcz_store.cc:276] 0806 04:07:43.874451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.875494 (+ 1043us) service_pool.cc:227] Handling call
0806 04:07:43.875874 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:43.925047 (+ 49173us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:43.956410 (+ 31363us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:43.957074 (+ 664us) log.cc:842] Serialized 26 byte log entry
0806 04:07:44.032671 (+ 75597us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:44.032726 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:44.032802 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:45.456544 (+1423742us) spinlock_profiling.cc:243] Waited 9.12 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:45.716116 (+259572us) raft_consensus.cc:1671] finished
0806 04:07:45.718325 (+ 2209us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:45.718596 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9072,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":41343,"spinlock_wait_cycles":20972416,"thread_start_us":73164,"threads_started":1,"wal-append.queue_time_us":420610}
I20260806 04:07:45.720873 7371 raft_consensus.cc:3032] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:45.721277 7371 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:45.721767 7674 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.725929 7594 consensus_queue.cc:237] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:45.727485 7695 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260806 04:07:45.713327 5034 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
W20260806 04:07:45.688053 7684 log.cc:924] Time spent T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.078s user 0.004s sys 0.005s
I20260806 04:07:45.746492 4817 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:45.749127 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.754184 7693 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.753878 7030 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.754575 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.756735 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.754874 7527 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.757578 5035 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.758805 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.759356 7527 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.755705 5029 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:45.759565 5032 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.761312 7030 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.756991 5038 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:45.754987 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.763015 5026 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
W20260806 04:07:45.763926 5816 leader_election.cc:334] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.729s (SENT)
I20260806 04:07:45.764505 5816 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.764698 7603 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.766012 7603 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.768662 7335 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:45.761307 7586 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.722122 7371 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.775777 7371 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.665313 7691 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.780570 7691 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:45.755788 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.781433 5028 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:45.781795 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:45.761637 7655 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.782507 5034 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.787344 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.790575 7587 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:45.793988 5038 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.799633 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:45.803591 4882 leader_election.cc:334] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.582s (SENT)
I20260806 04:07:45.804244 4882 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.808588 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.809262 4930 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:07:45.809504 4924 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.811164 5032 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260806 04:07:45.812023 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 95508.3us
I20260806 04:07:45.813441 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:07:45.816655 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 238072us
I20260806 04:07:45.816787 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.817687 4928 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:45.817740 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.818926 5029 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:45.820344 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.823880 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:45.831061 4882 leader_election.cc:334] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.607s (SENT)
I20260806 04:07:45.832369 4882 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.823364 7691 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:45.834775 4768 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:45.804286 7671 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:45.834807 7669 raft_consensus.cc:2741] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 04:07:45.836097 7683 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:45.836551 5029 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:45.801613 7556 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.848721 5028 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:45.855551 5038 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:45.830404 6970 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:45.863330 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.863665 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.863330 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.864568 4928 raft_consensus.cc:2354] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:45.865361 4924 raft_consensus.cc:2379] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.866761 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.867017 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.867496 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.867765 4882 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:45.866233 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:45.867659 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.869240 7643 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:45.869555 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:45.870479 4930 raft_consensus.cc:2354] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:07:45.870554 4927 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.872193 4926 raft_consensus.cc:2354] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:45.872778 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.873574 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:45.874284 4923 raft_consensus.cc:2354] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:45.860124 7030 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:45.876118 4996 leader_election.cc:334] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.884s (SENT)
I20260806 04:07:45.880125 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.883469 5028 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.852051 7691 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.885977 4882 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:45.878942 7669 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:45.886556 5038 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.754791 7693 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.892032 5032 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.893034 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.895538 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.896448 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.893034 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.899168 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.899948 5026 raft_consensus.cc:2362] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
I20260806 04:07:45.901700 4876 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.884160 7701 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:07:45.912034 4996 leader_election.cc:334] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.978s (SENT)
I20260806 04:07:45.912672 4996 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.893265 6970 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:45.915066 5028 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:45.917068 7419 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.920127 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:45.890318 7669 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:45.922762 4926 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:45.923105 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.925987 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.924508 7702 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:45.924430 7371 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:07:45.895223 7030 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:45.927063 5026 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
W20260806 04:07:45.933771 5816 leader_election.cc:334] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.526s (SENT)
I20260806 04:07:45.926015 4876 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:45.922802 6970 raft_consensus.cc:686] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:45.931972 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.925870 5029 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:45.936590 5816 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.911866 7701 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:45.943426 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:45.945252 5032 raft_consensus.cc:2362] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
W20260806 04:07:45.945604 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 52223.7us
I20260806 04:07:45.947561 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:45.948196 4996 leader_election.cc:334] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.075s (SENT)
I20260806 04:07:45.948659 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.945560 7701 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:45.943749 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:45.950016 4996 leader_election.cc:334] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.097s (SENT)
I20260806 04:07:45.950277 5038 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:45.950453 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.951272 7511 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:07:45.951588 4996 leader_election.cc:334] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.106s (SENT)
I20260806 04:07:45.951987 4996 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.952149 7511 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:07:45.952965 4996 leader_election.cc:334] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.147s (SENT)
I20260806 04:07:45.952989 7511 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:45.948800 5028 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:45.953403 4996 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:45.943609 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 52730.3us
I20260806 04:07:45.928843 7669 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:45.963091 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:45.961886 6970 consensus_queue.cc:237] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:45.921322 7683 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260806 04:07:45.968040 7655 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 04:07:45.969228 7705 log.cc:924] Time spent T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 04:07:45.969363 7683 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:45.969830 7683 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:45.970175 7683 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:45.928580 7690 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:45.940845 4817 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:45.974177 5038 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:45.971464 4817 raft_consensus.cc:595] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:45.945147 7643 raft_consensus.cc:686] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:45.991331 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.991187 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:45.993920 5028 raft_consensus.cc:2362] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
I20260806 04:07:45.930151 7527 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 04:07:45.993330 5038 raft_consensus.cc:2362] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
I20260806 04:07:46.009585 5037 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 04:07:46.011356 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 89644.4us
I20260806 04:07:46.011535 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:46.013058 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 60134.9us
W20260806 04:07:46.016676 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 90734.9us
I20260806 04:07:46.018596 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.019232 5038 raft_consensus.cc:2362] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4. Re-sending same reply.
I20260806 04:07:46.017522 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.018867 5816 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:45.997766 7511 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:46.023509 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:46.024401 4996 leader_election.cc:334] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.157s (SENT)
I20260806 04:07:46.027185 4996 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.029537 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.030355 5038 raft_consensus.cc:2362] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4. Re-sending same reply.
I20260806 04:07:45.970711 7609 consensus_queue.cc:1046] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260806 04:07:46.038092 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.014354 7030 raft_consensus.cc:686] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:46.039435 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 356300us
I20260806 04:07:46.040623 4813 raft_consensus.cc:2379] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:46.025640 7586 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:46.018522 7556 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.042963 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.047894 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.048451 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.049991 4808 raft_consensus.cc:2354] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.032369 7690 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.018221 7710 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:46.049401 5038 raft_consensus.cc:2362] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
I20260806 04:07:46.052556 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.053634 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:46.055968 4996 leader_election.cc:334] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 2.424s (SENT)
I20260806 04:07:46.056743 7556 consensus_queue.cc:1046] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:46.057596 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.057932 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.056932 7671 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:46.058804 4809 raft_consensus.cc:2354] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.061200 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.048995 7586 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:46.071815 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.072438 4808 raft_consensus.cc:2362] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:46.072031 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.073289 4816 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:46.073482 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.063849 4992 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.055399 7371 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:46.035331 7587 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.017s
I20260806 04:07:46.069413 7419 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.070107 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.070997 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.071214 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.058774 7710 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:46.117409 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 61490.7us
I20260806 04:07:46.110090 7371 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:46.117550 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.122485 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.123818 4818 raft_consensus.cc:1261] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:46.124007 4813 raft_consensus.cc:1261] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:46.124406 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.051577 7655 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260806 04:07:46.122444 7371 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:46.129199 7683 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.119204 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:46.131160 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 71458.3us
I20260806 04:07:46.059577 7691 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:46.132318 4809 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:46.132871 4996 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:46.115590 4804 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:46.117422 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.135201 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.136574 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.138044 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.138844 4808 raft_consensus.cc:2362] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3. Re-sending same reply.
I20260806 04:07:46.143872 5028 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:46.108517 7586 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:46.155437 7723 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:46.158573 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 122424us
W20260806 04:07:46.159979 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 134355us
I20260806 04:07:46.167130 4816 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:46.170966 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.174933 5038 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:46.165966 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.176405 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.108251 7714 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:46.118494 7690 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.177276 5034 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:46.194271 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.186564 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.117700 7527 raft_consensus.cc:3032] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:46.195966 5029 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:46.186893 5037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
mode: GRACEFUL
new_leader_uuid: "22fe557f852d4415a96fd683f970cc24"
from {username='slave'} at 127.0.0.1:39956
I20260806 04:07:46.196247 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.196843 5037 raft_consensus.cc:595] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Received request to transfer leadership to TS 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:46.197055 5036 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:46.197198 5037 raft_consensus.cc:603] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:46.115680 7693 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.184971 4882 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.144019 7511 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.186534 5032 raft_consensus.cc:1261] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:46.171623 7652 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:46.072063 7643 consensus_queue.cc:237] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.236694 5038 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:46.183753 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.240540 4809 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:46.186246 7691 raft_consensus.cc:2741] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 04:07:46.186167 7678 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:46.201821 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 184352us
I20260806 04:07:46.204767 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.253643 5034 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:46.213939 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.255214 5026 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:46.214926 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.217010 4920 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:46.217404 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:46.219034 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.257309 5032 raft_consensus.cc:2362] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3. Re-sending same reply.
W20260806 04:07:46.202378 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 154194us
W20260806 04:07:46.207046 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 74237.4us
W20260806 04:07:46.216199 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 104285us
I20260806 04:07:46.190508 7702 raft_consensus.cc:686] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:46.249876 5029 raft_consensus.cc:2396] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260806 04:07:46.255121 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 70245.5us
I20260806 04:07:46.258916 5816 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.255964 5035 raft_consensus.cc:2362] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3. Re-sending same reply.
I20260806 04:07:46.268877 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.195596 7419 consensus_queue.cc:1046] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 04:07:46.205942 7527 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:46.282836 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.283578 5026 raft_consensus.cc:2362] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3. Re-sending same reply.
I20260806 04:07:46.218132 7714 raft_consensus.cc:686] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:46.278463 4992 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.285881 4992 leader_election.cc:398] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.286199 4930 raft_consensus.cc:1226] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:46.274863 4996 leader_election.cc:398] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.287072 4996 leader_election.cc:401] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 04:07:46.286176 4992 leader_election.cc:401] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 04:07:46.184635 7701 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.292915 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:46.288237 4920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:46.296711 4920 raft_consensus.cc:595] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Received request to transfer leadership to TS d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.297180 4920 raft_consensus.cc:603] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:46.300488 5863 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:46.263959 6970 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:46.303647 4926 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:46.310184 4992 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.316380 4930 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:46.321074 4992 leader_election.cc:398] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:07:46.322345 4992 leader_election.cc:398] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.323545 4768 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.326473 4992 leader_election.cc:398] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.327554 4992 leader_election.cc:401] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 04:07:46.192425 7723 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:46.341440 4657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 223326us
I20260806 04:07:46.384861 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:46.298269 7691 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:46.383033 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 59597.8us
I20260806 04:07:46.209151 7030 consensus_queue.cc:237] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.309926 7683 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.312611 7511 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.270160 7739 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.166s
I20260806 04:07:46.400990 4930 raft_consensus.cc:1226] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
W20260806 04:07:46.397980 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 84390.7us
I20260806 04:07:46.402968 4882 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.384207 7609 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:46.404270 7711 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:46.312189 7727 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.227s
I20260806 04:07:46.394291 7316 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.398298 7737 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.411162 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:46.463471 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 60614.6us
I20260806 04:07:46.412914 7702 consensus_queue.cc:237] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.275255 7678 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.394601 7527 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.397956 4919 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 04:07:46.330062 6970 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.420487 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:46.429411 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.390062 7521 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:46.384143 7690 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
W20260806 04:07:46.440673 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 114379us
W20260806 04:07:46.472849 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 182825us
I20260806 04:07:46.477692 4992 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.441773 7721 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:46.442516 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 04:07:46.448138 7643 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:46.482005 4992 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:46.483594 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 54300.1us
W20260806 04:07:46.463609 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 151114us
I20260806 04:07:46.413172 7713 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:46.415931 7671 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:46.267109 7710 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.471531 4882 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:46.460335 7683 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.461411 7609 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:46.478720 5816 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.466501 7316 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.433511 7371 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.486562 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.494150 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:46.465453 7731 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20260806 04:07:46.476157 7693 consensus_queue.cc:1159] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "99690415a7b3441fbdb5e943667556a4" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 3 } }
I20260806 04:07:46.524299 7723 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:46.536403 7371 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:46.525652 4657 catalog_manager.cc:4584] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): ChangeConfig:REMOVE_PEER RPC failed for tablet 086f8af9433f44aa8aa4fccbfa90eb02: Remote error: Service unavailable: leader transfer in progress
I20260806 04:07:46.524300 7678 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:46.523293 7701 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.537420 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:46.538218 7742 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260806 04:07:46.482875 7511 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.465572 7746 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260806 04:07:46.546523 7701 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:46.547133 7701 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.526438 5026 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:46.533100 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.553947 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.538416 5862 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:46.484400 7527 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:46.563611 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 84995.8us
W20260806 04:07:46.577375 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 90899.5us
I20260806 04:07:46.578428 4996 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.491397 7713 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:46.534147 7521 raft_consensus.cc:979] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: : Instructing follower 99690415a7b3441fbdb5e943667556a4 to start an election
W20260806 04:07:46.531832 7750 log.cc:924] Time spent T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260806 04:07:46.550479 7603 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.784s user 0.011s sys 0.002s
I20260806 04:07:46.591845 4996 leader_election.cc:398] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.461225 7711 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.525328 6970 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:46.637789 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 175904us
I20260806 04:07:46.639593 5030 raft_consensus.cc:2396] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 04:07:46.640484 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 143445us
I20260806 04:07:46.530627 7727 raft_consensus.cc:2781] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:46.492655 7671 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:46.605914 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:46.607568 5863 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:46.600595 7713 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:46.523851 7740 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 04:07:46.596889 7511 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:46.610173 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:46.484313 7721 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:46.620283 7691 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.610342 7587 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.622009 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:46.626101 7594 raft_consensus.cc:3032] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:46.630823 7521 raft_consensus.cc:1067] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Signalling peer 99690415a7b3441fbdb5e943667556a4 to start an election
I20260806 04:07:46.597790 7762 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.601835 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.530371 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:46.670281 4814 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:46.565560 7746 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:46.485523 7643 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:46.677181 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 66574.5us
I20260806 04:07:46.714131 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.674991 4882 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:46.601612 7678 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:46.526955 7586 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.621521 4674 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56998 (request call id 142) took 2598 ms. Trace:
I20260806 04:07:46.461127 7714 consensus_queue.cc:237] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.644227 4669 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 2 to 3, leader changed from d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 3 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:46.599025 7701 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.490698 7690 consensus_queue.cc:1046] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.186s
I20260806 04:07:46.675854 7510 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260806 04:07:46.670542 7727 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:46.655440 7661 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:46.682708 7511 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:46.720119 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 118363us
I20260806 04:07:46.703275 7766 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:46.722293 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.722954 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.728842 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.756064 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.671958 5862 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:46.756418 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.534920 7609 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:46.757051 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.755018 4814 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:46.693017 7594 raft_consensus.cc:729] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:46.760375 4804 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:46.731443 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:46.683561 7683 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260806 04:07:46.759140 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.760295 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:46.731134 5812 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:46.748924 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.751961 4669 catalog_manager.cc:5495] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:07:46.761289 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:46.828246 4809 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:46.754449 4996 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.763489 4882 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:46.758174 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:46.757251 7655 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:46.757839 7653 consensus_queue.cc:1046] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260806 04:07:46.767995 7759 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260806 04:07:46.757450 7511 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:46.757839 7727 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:46.720086 7762 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.792900 4818 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:46.808894 4814 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:46.750064 7746 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.730018 7030 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:46.710093 7723 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260806 04:07:46.756512 7724 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.798508 4810 raft_consensus.cc:2354] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:07:46.828280 4669 catalog_manager.cc:5495] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
W20260806 04:07:46.829449 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 66029.6us
W20260806 04:07:46.814158 7768 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
W20260806 04:07:46.834553 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 103551us
I20260806 04:07:46.835825 4804 raft_consensus.cc:1261] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:46.843640 7521 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:07:46.844334 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 84131.1us
I20260806 04:07:46.872288 4768 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:46.844808 7734 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:46.750563 4674 rpcz_store.cc:276] 0806 04:07:44.023164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:44.023458 (+ 294us) service_pool.cc:227] Handling call
0806 04:07:46.621422 (+2597964us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:45.001709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:45.095137 (+ 93428us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:45.095193 (+ 56us) write_op.cc:435] Acquired schema lock
0806 04:07:45.095210 (+ 17us) tablet.cc:660] Decoding operations
0806 04:07:45.100105 (+ 4895us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:45.100259 (+ 154us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:45.100300 (+ 41us) tablet.cc:683] Acquiring locks for 10 operations
0806 04:07:45.230764 (+130464us) tablet.cc:699] Row locks acquired
0806 04:07:45.230790 (+ 26us) write_op.cc:260] PREPARE: finished
0806 04:07:45.230984 (+ 194us) write_op.cc:270] Start()
0806 04:07:45.231152 (+ 168us) write_op.cc:276] Timestamp: P: 1785989265230912 usec, L: 0
0806 04:07:45.231191 (+ 39us) op_driver.cc:331] REPLICATION: starting
0806 04:07:45.231955 (+ 764us) log.cc:842] Serialized 3946 byte log entry
0806 04:07:45.830612 (+598657us) op_driver.cc:444] REPLICATION: finished
0806 04:07:45.962774 (+132162us) write_op.cc:301] APPLY: starting
0806 04:07:45.978247 (+ 15473us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:46.042190 (+ 63943us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:46.059079 (+ 16889us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:46.390456 (+331377us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:46.411155 (+ 20699us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0806 04:07:46.411206 (+ 51us) write_op.cc:312] APPLY: finished
0806 04:07:46.411587 (+ 381us) log.cc:842] Serialized 77 byte log entry
0806 04:07:46.411820 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:46.411994 (+ 174us) write_op.cc:454] Released schema lock
0806 04:07:46.412063 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:46.412388 (+ 325us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:46.412418 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":94202,"apply.run_cpu_time_us":6297,"apply.run_wall_time_us":471262,"num_ops":10,"prepare.queue_time_us":176324,"prepare.run_cpu_time_us":18342,"prepare.run_wall_time_us":322093,"raft.queue_time_us":412844,"raft.run_cpu_time_us":16778,"raft.run_wall_time_us":73036,"replication_time_us":596335,"thread_start_us":339841,"threads_started":4,"wal-append.queue_time_us":77153}]]}
I20260806 04:07:46.825932 7731 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:46.834088 7674 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:46.730208 7671 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.876463 4804 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 04:07:46.880117 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.760617 7766 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:46.851527 7030 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:46.851023 7690 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:46.878598 4818 raft_consensus.cc:1226] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260806 04:07:46.873795 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 119432us
I20260806 04:07:46.886762 4996 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.888468 4996 leader_election.cc:398] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.888839 4996 leader_election.cc:401] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 04:07:46.759989 7754 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.107s
I20260806 04:07:46.759832 7661 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:46.905862 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:46.871796 7521 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:46.896965 7316 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:46.875530 7734 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:46.879523 7674 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:46.913281 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.885120 7030 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:46.890879 7746 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:46.923768 4996 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.924930 4996 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:46.891250 7726 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:46.903538 7762 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:07:46.908337 7609 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:46.928119 7759 raft_consensus.cc:2781] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:46.933871 7771 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 04:07:46.883896 7693 consensus_queue.cc:1035] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260806 04:07:46.934248 7781 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20260806 04:07:46.935111 7784 log.cc:924] Time spent T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260806 04:07:46.931418 7726 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:46.902148 7724 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260806 04:07:46.943879 7759 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:46.936659 7762 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:46.903373 7739 consensus_queue.cc:1046] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260806 04:07:46.937348 7586 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.931243 7335 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260806 04:07:46.919252 7674 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:46.915792 7521 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:46.927389 4814 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:46.860944 7755 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260806 04:07:46.876014 7720 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.240s
I20260806 04:07:46.919363 7683 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:46.911113 7743 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.171s
I20260806 04:07:46.949388 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:47.056063 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 106810us
I20260806 04:07:47.057749 4768 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:46.918936 7734 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:46.918215 7557 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:46.945765 7669 consensus_meta.cc:287] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.956s user 0.009s sys 0.004s
I20260806 04:07:46.926571 7690 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:07:46.928735 7767 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:47.073931 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1243) took 1026 ms. Trace:
I20260806 04:07:46.936026 7643 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:46.862102 7594 consensus_queue.cc:260] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:46.926152 5853 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:46.952677 7711 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:47.049363 4882 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:47.041837 7759 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:46.945822 7609 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.040920 7721 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:47.051648 7371 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.074396 4810 rpcz_store.cc:276] 0806 04:07:46.047161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.047316 (+ 155us) service_pool.cc:227] Handling call
0806 04:07:47.073702 (+1026386us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b540011dd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:47.073857 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1703319808}
W20260806 04:07:47.192737 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 135057us
I20260806 04:07:47.011584 7749 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260806 04:07:47.063437 7723 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:47.255839 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 270) took 1092 ms. Trace:
W20260806 04:07:47.256397 7783 log.cc:924] Time spent T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.245s user 0.000s sys 0.000s
I20260806 04:07:47.288412 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:47.075688 7746 raft_consensus.cc:686] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:47.287120 4808 rpcz_store.cc:276] 0806 04:07:46.163532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.163711 (+ 179us) service_pool.cc:227] Handling call
0806 04:07:46.164234 (+ 523us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.459969 (+295735us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.459998 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:46.462550 (+ 2552us) log.cc:842] Serialized 26 byte log entry
0806 04:07:46.558817 (+ 96267us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:46.558861 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:46.616067 (+ 57206us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400021554. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:46.616118 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:47.153677 (+537559us) raft_consensus.cc:1671] finished
0806 04:07:47.250090 (+ 96413us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:47.255691 (+ 5601us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5623,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":4767,"spinlock_wait_cycles":604329344,"thread_start_us":78759,"threads_started":1,"wal-append.queue_time_us":230527}
I20260806 04:07:47.153000 7762 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:47.213924 7709 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:47.061861 7683 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:47.318693 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 286) took 1011 ms. Trace:
I20260806 04:07:47.280910 7594 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:47.319898 4928 rpcz_store.cc:276] 0806 04:07:46.306935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.307082 (+ 147us) service_pool.cc:227] Handling call
0806 04:07:46.614355 (+307273us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:46.614560 (+ 205us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.631167 (+ 16607us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.631233 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:46.631822 (+ 589us) log.cc:842] Serialized 26 byte log entry
0806 04:07:46.713492 (+ 81670us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:46.713532 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:46.713603 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:47.295882 (+582279us) raft_consensus.cc:1671] finished
0806 04:07:47.295906 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:47.318599 (+ 22693us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1392,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":154,"spinlock_wait_cycles":2957696,"thread_start_us":81274,"threads_started":1,"wal-append.queue_time_us":166920}
I20260806 04:07:47.285459 7752 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.324151 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:47.287171 7721 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.065972 7669 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.347836 4810 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:47.313956 7739 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:47.321673 7587 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.359993 5816 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:47.364847 4924 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:47.286192 7371 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.346410 7661 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:47.348809 7723 raft_consensus.cc:686] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:47.370894 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 82562.3us
I20260806 04:07:47.419755 4768 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:47.407339 7774 log.cc:924] Time spent T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
W20260806 04:07:47.514110 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 94435.9us
W20260806 04:07:47.413913 7527 consensus_meta.cc:287] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.834s user 0.009s sys 0.003s
I20260806 04:07:47.370779 7643 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.370527 7711 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:47.367468 7739 raft_consensus.cc:686] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:47.573846 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 146595us
I20260806 04:07:47.559751 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:47.345912 7762 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:47.584159 4657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 67425.8us
I20260806 04:07:47.421123 7690 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.366250 7761 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.358179 7721 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.424814 7661 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:47.371759 7587 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:47.572574 7557 raft_consensus.cc:2741] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Leader pre-election decision vote started in defunct term 2: won
W20260806 04:07:47.688833 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 83079.9us
I20260806 04:07:47.573676 7724 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.353041 7752 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:47.699223 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 140022us
W20260806 04:07:47.643002 7710 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.872s user 0.023s sys 0.040s
I20260806 04:07:47.697793 7711 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:47.712577 7661 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:47.615870 7643 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.840106 7743 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.840579 7683 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:47.761533 7790 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.606s user 0.002s sys 0.000s
I20260806 04:07:47.841800 7669 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:47.794859 7710 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.024s user 0.025s sys 0.040s
W20260806 04:07:47.817579 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51222us
I20260806 04:07:47.829465 7794 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:47.843441 7810 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.188s user 0.001s sys 0.002s
I20260806 04:07:47.830709 7739 consensus_queue.cc:237] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:47.702172 7690 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:47.747048 7371 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:47.807389 7609 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:47.417783 7746 consensus_queue.cc:237] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:47.840709 7743 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:47.841976 7796 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20260806 04:07:47.831373 7762 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:47.865324 7787 log.cc:924] Time spent T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.677s user 0.000s sys 0.003s
I20260806 04:07:47.789760 7724 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.843870 7794 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:47.717499 7761 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:47.876749 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:47.960213 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 279) took 1029 ms. Trace:
I20260806 04:07:47.882593 4928 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:47.746520 7821 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.885637 7669 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.900686 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1968) took 1176 ms. Trace:
W20260806 04:07:47.901904 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.994s user 0.016s sys 0.003s
W20260806 04:07:47.978432 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.071s user 0.017s sys 0.004s
I20260806 04:07:47.808166 7419 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.845067 7661 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:47.878325 7587 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:47.872291 7752 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:47.871882 7643 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:47.979456 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.07208e+06us
I20260806 04:07:47.947322 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:47.991309 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:47.886548 4919 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 04:07:47.837850 7775 log.cc:924] Time spent T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.006s user 0.006s sys 0.017s
I20260806 04:07:47.870527 7721 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:48.060483 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 73016.2us
I20260806 04:07:48.009552 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:47.995920 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:48.015087 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 51667us
W20260806 04:07:47.999415 7727 consensus_meta.cc:287] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.141s user 0.011s sys 0.008s
W20260806 04:07:47.934777 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.011s sys 0.017s
W20260806 04:07:48.087754 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.833s user 0.012s sys 0.017s
W20260806 04:07:48.015793 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.009s sys 0.021s
I20260806 04:07:47.961421 4804 rpcz_store.cc:276] 0806 04:07:46.917438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.917609 (+ 171us) service_pool.cc:227] Handling call
0806 04:07:46.917947 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.926788 (+ 8841us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.926817 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:46.944367 (+ 17550us) log.cc:842] Serialized 26 byte log entry
0806 04:07:47.308997 (+364630us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:47.309055 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:47.309157 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:47.946088 (+636931us) raft_consensus.cc:1671] finished
0806 04:07:47.946118 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:47.946322 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9405,"raft.queue_time_us":12790,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":10108,"thread_start_us":264508,"threads_started":1,"wal-append.queue_time_us":404256}
I20260806 04:07:47.964457 7371 consensus_queue.cc:1046] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.101s
I20260806 04:07:47.975507 4809 rpcz_store.cc:276] 0806 04:07:46.724772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.836163 (+111391us) service_pool.cc:227] Handling call
0806 04:07:46.836507 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.842086 (+ 5579us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.842120 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:46.842617 (+ 497us) log.cc:842] Serialized 26 byte log entry
0806 04:07:46.896833 (+ 54216us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:46.896874 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:46.896945 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:47.900460 (+1003515us) raft_consensus.cc:1671] finished
0806 04:07:47.900486 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:47.900621 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5193,"raft.queue_time_us":5479,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":139,"thread_start_us":53457,"threads_started":1,"wal-append.queue_time_us":150857}
W20260806 04:07:48.022832 7818 log.cc:924] Time spent T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.139s user 0.004s sys 0.000s
I20260806 04:07:48.040201 7419 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.185398 7419 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:48.185935 7030 consensus_meta.cc:287] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.243s user 0.016s sys 0.007s
W20260806 04:07:48.186497 7511 consensus_meta.cc:287] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.330s user 0.013s sys 0.008s
I20260806 04:07:48.187001 7511 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.152091 7816 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.189282 7511 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.189239 7816 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:48.194365 7678 consensus_meta.cc:287] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.445s user 0.013s sys 0.008s
I20260806 04:07:48.213083 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.150872 4928 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:48.216513 4657 catalog_manager.cc:4584] TS df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): ChangeConfig:REMOVE_PEER RPC failed for tablet 086f8af9433f44aa8aa4fccbfa90eb02: Remote error: Service unavailable: leader transfer in progress
W20260806 04:07:48.216817 7713 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.529s user 0.020s sys 0.002s
W20260806 04:07:48.152068 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 142830us
I20260806 04:07:48.217409 7713 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.219195 5812 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:48.044217 7587 consensus_queue.cc:1046] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20260806 04:07:48.220850 7557 raft_consensus.cc:3032] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:48.221045 5812 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:07:48.166996 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.241s user 0.018s sys 0.027s
I20260806 04:07:48.221335 7557 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:48.222129 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.29714e+06us
I20260806 04:07:47.912643 7723 consensus_queue.cc:237] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.222352 7557 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.223886 7557 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:48.022547 7798 consensus_queue.cc:1046] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.143s
W20260806 04:07:48.174577 7521 consensus_meta.cc:287] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.115s user 0.014s sys 0.006s
I20260806 04:07:48.171140 7800 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.170800 7371 raft_consensus.cc:3032] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:48.226332 7371 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:48.226418 7800 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.227380 7371 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.228806 7371 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:48.171548 7797 consensus_queue.cc:1046] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260806 04:07:48.229662 7308 raft_consensus.cc:3032] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:48.230067 7308 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:48.175457 4926 raft_consensus.cc:1261] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 04:07:48.230523 7594 consensus_meta.cc:287] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.881s user 0.019s sys 0.000s
I20260806 04:07:48.231001 7308 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:48.179869 7817 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260806 04:07:48.232307 7308 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:48.232690 7800 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:48.180900 7806 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.462s user 0.003s sys 0.000s
I20260806 04:07:48.233744 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.234517 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:48.197074 7661 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.235316 4923 raft_consensus.cc:2379] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:48.236604 7661 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.236953 7691 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.237227 7661 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 04:07:48.237478 4996 connection.cc:232] RPC call timeout handler was delayed by -1.3356284501496702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
I20260806 04:07:48.237813 7661 raft_consensus.cc:686] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:48.238173 7652 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:48.239392 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.182380 7761 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.200052 7816 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:48.176815 7789 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.225s user 0.011s sys 0.000s
I20260806 04:07:47.956660 7762 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.210337 7731 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:07:48.205251 7799 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.376s user 0.001s sys 0.000s
I20260806 04:07:48.241055 7661 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.244609 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.237504 7691 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.247443 7737 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.247576 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.186422 7030 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:48.189941 7759 consensus_meta.cc:287] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.878s user 0.017s sys 0.000s
I20260806 04:07:48.180927 7794 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.213372 7678 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:48.200033 7690 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:48.224530 7734 consensus_meta.cc:287] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.149s user 0.012s sys 0.008s
I20260806 04:07:48.225514 7798 raft_consensus.cc:3032] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:48.155110 7710 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:48.231818 7586 raft_consensus.cc:3032] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:48.234678 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 576) took 1328 ms. Trace:
I20260806 04:07:48.318816 5862 rpcz_store.cc:276] 0806 04:07:46.906086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.906276 (+ 190us) service_pool.cc:227] Handling call
0806 04:07:46.926279 (+ 20003us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.932661 (+ 6382us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.932691 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:46.949979 (+ 17288us) log.cc:842] Serialized 26 byte log entry
0806 04:07:47.318232 (+368253us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:47.318279 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:47.318347 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:48.234383 (+916036us) raft_consensus.cc:1671] finished
0806 04:07:48.234402 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:48.234547 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":328975,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":216,"thread_start_us":274050,"threads_started":1,"wal-append.queue_time_us":405281}
W20260806 04:07:48.179322 7316 consensus_meta.cc:287] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.258s user 0.018s sys 0.004s
I20260806 04:07:48.026528 7821 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.239418 4926 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:48.012434 7669 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.235348 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:48.089253 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.03985e+06us
I20260806 04:07:48.244848 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.392136 4817 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:48.244669 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1977) took 1375 ms. Trace:
I20260806 04:07:48.305868 7511 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:47.991005 7609 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.284317 7587 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260806 04:07:48.249408 4768 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:48.255669 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:48.416381 4804 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:48.270020 4927 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:48.272825 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:48.140700 7643 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:07:48.417968 4810 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.003s sys 0.056s
I20260806 04:07:48.418193 5863 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:48.301909 4928 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
W20260806 04:07:48.210381 7674 consensus_meta.cc:287] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.151s user 0.013s sys 0.005s
I20260806 04:07:48.314139 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.423009 5026 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:48.246044 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.317451 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:48.269134 7691 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.245587 7731 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:48.390820 7316 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:48.429555 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.430115 5867 raft_consensus.cc:2396] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:48.328962 5038 raft_consensus.cc:1261] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:48.331175 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.304519 7816 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.393363 4816 rpcz_store.cc:276] 0806 04:07:46.869486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.894536 (+ 25050us) service_pool.cc:227] Handling call
0806 04:07:46.916231 (+ 21695us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:47.292134 (+375903us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:47.292218 (+ 84us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:47.292967 (+ 749us) log.cc:842] Serialized 26 byte log entry
0806 04:07:47.557961 (+264994us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:47.783250 (+225289us) log.cc:842] Serialized 14 byte log entry
0806 04:07:47.784376 (+ 1126us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:48.240899 (+456523us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54000ffd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:48.240957 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:48.241008 (+ 51us) raft_consensus.cc:1671] finished
0806 04:07:48.244461 (+ 3453us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:48.244609 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":250623,"prepare.queue_time_us":370386,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":139,"raft.queue_time_us":233544,"raft.run_cpu_time_us":568,"raft.run_wall_time_us":31682,"spinlock_wait_cycles":342652160,"thread_start_us":328765,"threads_started":2,"wal-append.queue_time_us":615133}
I20260806 04:07:48.436794 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.244010 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:48.439695 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.303951 7594 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.339068 7678 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:48.322299 7690 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.244879 7762 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.401480 5035 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:48.471837 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:48.401475 5036 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 04:07:48.248031 7737 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.472681 7762 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.473555 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.474025 7739 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:48.474234 5036 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:48.474457 7737 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.404598 4931 raft_consensus.cc:1226] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:48.474603 7739 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.475260 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.475643 7669 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.475925 5036 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:48.476711 7739 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.476904 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.477505 5036 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:48.255147 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:48.311038 7798 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:48.418313 4810 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.507s user 0.003s sys 0.056s
I20260806 04:07:48.121594 7727 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.303467 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.314632 7734 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.496838 4804 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:48.499498 5816 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:48.500936 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 245888us
I20260806 04:07:48.503086 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.423309 7609 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.506484 7737 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:48.506743 7808 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.223s
I20260806 04:07:48.507337 7737 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:48.508462 7829 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:48.509110 7829 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.510524 7737 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:48.510143 7678 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.454313 7691 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:48.415153 7511 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.332442 7710 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.426616 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:48.428169 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 178850us
I20260806 04:07:48.428673 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.303316 7761 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.267724 7521 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.433135 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.433386 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.438454 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.451547 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:48.451983 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.399115 7743 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.460119 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.318598 7586 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:48.461141 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.459339 7594 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.465363 7831 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:48.466789 7776 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.635s user 0.004s sys 0.014s
I20260806 04:07:48.468902 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:48.483290 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.485702 7762 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.308363 7759 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.431385 7674 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.433864 7643 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.453928 7816 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.510948 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:48.841327 7557 consensus_meta.cc:287] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.617s user 0.012s sys 0.000s
I20260806 04:07:48.912577 5867 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:48.831200 7829 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.913502 7730 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:48.914621 5028 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:48.914125 7730 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.914892 4817 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:48.915890 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.916606 4817 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:48.917073 7730 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.917403 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:48.832932 7727 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.918035 7730 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:48.458832 7316 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.918535 7831 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.919064 7832 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.919165 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:48.919759 7832 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.920727 5034 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:48.921908 4813 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:48.921998 7832 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.512233 7030 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.922617 7832 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:48.923393 7832 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.924587 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1537) took 2650 ms. Trace:
I20260806 04:07:48.925482 7832 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.918877 7316 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.706853 7724 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.926921 7761 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.924968 4925 rpcz_store.cc:276] 0806 04:07:46.274244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.274850 (+ 606us) service_pool.cc:227] Handling call
0806 04:07:46.303583 (+ 28733us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:07:46.430930 (+127347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.430977 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:07:46.431705 (+ 728us) log.cc:842] Serialized 52 byte log entry
0806 04:07:46.670754 (+239049us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:07:46.871762 (+201008us) log.cc:842] Serialized 14 byte log entry
0806 04:07:46.873185 (+ 1423us) log.cc:842] Serialized 14 byte log entry
0806 04:07:46.875852 (+ 2667us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:46.875937 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:48.465786 (+1589849us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:48.924370 (+458584us) raft_consensus.cc:1671] finished
0806 04:07:48.924392 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:48.924521 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83454,"prepare.queue_time_us":96924,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":7664,"raft.queue_time_us":85570,"raft.run_cpu_time_us":499,"raft.run_wall_time_us":16778,"spinlock_wait_cycles":10030464,"thread_start_us":428526,"threads_started":3,"wal-append.queue_time_us":259423}
I20260806 04:07:48.927476 7762 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.929165 7761 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.929788 7762 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.918556 7730 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.511293 7737 raft_consensus.cc:686] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:48.930815 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:48.931526 7816 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.931664 7737 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:48.932335 7730 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.825584 7848 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.705730 7833 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:48.929354 7691 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.923776 7521 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.933768 7833 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.932935 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:48.934913 5863 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:48.935989 4995 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:48.936043 7674 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.936149 5037 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:48.937114 7833 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.937458 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:48.938391 7643 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.693614 4930 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:48.921066 7710 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.586627 7819 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.938304 7586 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.921125 7831 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.939363 7819 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.940187 7586 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:48.585184 7798 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:48.940778 7594 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.722975 7839 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:48.585904 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.941927 7819 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.927899 7316 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.942430 4808 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:48.942361 7839 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:48.592165 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 90847.7us
I20260806 04:07:48.943854 5816 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:48.945549 7839 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.476830 7713 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.946559 5026 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:48.947224 5038 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:48.948189 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:48.668969 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.951357 7759 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.952188 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.953299 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:48.953260 7848 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.020710 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.943588 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:49.024310 4808 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:48.914029 4816 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
W20260806 04:07:49.032723 4923 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.030s sys 0.064s
W20260806 04:07:49.033262 4923 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.030s sys 0.064s
W20260806 04:07:49.054644 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 115109us
I20260806 04:07:48.922704 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:48.958359 5032 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:48.960194 4920 raft_consensus.cc:2379] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:48.960354 7511 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:48.488545 7690 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:48.961264 7652 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.713s user 0.006s sys 0.010s
I20260806 04:07:48.511925 7830 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:48.918551 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:48.962757 7586 consensus_queue.cc:1035] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:48.963430 7710 consensus_queue.cc:1035] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:48.965603 7829 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:48.966820 5035 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:07:48.968134 7678 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:48.968439 5862 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:48.974035 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:48.974375 4926 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:48.975684 5859 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260806 04:07:48.975795 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:48.965912 5029 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.007s sys 0.008s
I20260806 04:07:48.513851 7734 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:48.947889 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:48.983492 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:49.008306 7845 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:49.017627 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 81749us
I20260806 04:07:48.511713 7836 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:48.938644 5037 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:49.028438 5816 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:49.062801 5034 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:49.063342 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:49.061703 5028 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:48.911017 7609 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:49.071218 4808 raft_consensus.cc:1261] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:48.979928 7721 raft_consensus.cc:2741] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260806 04:07:49.194134 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:49.426620 4996 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34042 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 11 similar messages]
I20260806 04:07:49.526292 5026 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:49.374846 4876 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.512s (SENT) [suppressed 60 similar messages]
I20260806 04:07:49.335098 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 281) took 2044 ms. Trace:
I20260806 04:07:49.393203 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 285) took 3086 ms. Trace:
W20260806 04:07:49.516167 5029 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.008s sys 0.017s
W20260806 04:07:49.529002 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 154198us
I20260806 04:07:49.529143 4927 rpcz_store.cc:276] 0806 04:07:46.306505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.306635 (+ 130us) service_pool.cc:227] Handling call
0806 04:07:46.539944 (+233309us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:46.551556 (+ 11612us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:48.417511 (+1865955us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.392861 (+975350us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540016f3d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:49.393117 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3954920448}
I20260806 04:07:49.126926 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:49.380743 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:49.341086 4993 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.004s sys 0.024s
W20260806 04:07:49.531046 4993 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.106s user 0.007s sys 0.039s
I20260806 04:07:49.531129 4925 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:49.122570 4992 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:49.529080 4813 rpcz_store.cc:276] 0806 04:07:47.290791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:47.291154 (+ 363us) service_pool.cc:227] Handling call
0806 04:07:47.291564 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:48.922326 (+1630762us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.334801 (+412475us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54003d7a54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:49.335027 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3748997376}
I20260806 04:07:49.016920 7762 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:49.501896 4657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 474645us
I20260806 04:07:49.517892 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1979) took 1115 ms. Trace:
I20260806 04:07:49.334054 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 287) took 1091 ms. Trace:
W20260806 04:07:49.529940 4876 leader_election.cc:334] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.512s (SENT)
I20260806 04:07:49.073151 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:49.534515 4876 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:49.534389 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1986) took 1115 ms. Trace:
I20260806 04:07:49.533946 4809 rpcz_store.cc:276] 0806 04:07:48.242216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.253325 (+ 11109us) service_pool.cc:227] Handling call
0806 04:07:48.283828 (+ 30503us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:48.919984 (+636156us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:48.920014 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:48.920058 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:48.920093 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.333710 (+413617us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54003d8454. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:49.333780 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:49.333968 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1462546176}
I20260806 04:07:49.535626 4817 rpcz_store.cc:276] 0806 04:07:48.419153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.918291 (+499138us) service_pool.cc:227] Handling call
0806 04:07:49.534327 (+616036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:49.072046 5038 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:49.532604 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 410184us
I20260806 04:07:49.517402 4804 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:49.538166 4810 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.059s user 0.004s sys 0.011s
W20260806 04:07:49.538287 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 554859us
I20260806 04:07:49.074892 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:49.168849 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.195068 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:49.257274 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 61230.9us
W20260806 04:07:49.315497 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 75237us
I20260806 04:07:49.533342 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1599) took 1184 ms. Trace:
I20260806 04:07:49.536487 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1860) took 1022 ms. Trace:
W20260806 04:07:49.332321 7793 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 2.055s user 0.007s sys 0.000s
I20260806 04:07:49.540037 5028 rpcz_store.cc:276] 0806 04:07:48.334511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.451211 (+116700us) service_pool.cc:227] Handling call
0806 04:07:49.518555 (+1067344us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:49.524637 7652 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.269s user 0.009s sys 0.018s
I20260806 04:07:49.532691 5867 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:49.445996 7711 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.541399 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:49.135874 7371 consensus_meta.cc:287] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.907s user 0.013s sys 0.000s
I20260806 04:07:49.323257 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.145414 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.543179 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:49.587388 4813 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:49.389591 5813 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.000s sys 0.021s
W20260806 04:07:49.588142 5813 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.164s user 0.000s sys 0.034s
I20260806 04:07:49.519529 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.527504 4996 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34042 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 192)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1886)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1883)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1859)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1885)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1888)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1861)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1594)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:34050 (request call id 1595)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1596)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1601)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1602)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1607)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1852)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1856)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1857)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1862)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1866)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1870)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1872)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1873)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1875)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 186)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 189)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1891)
I20260806 04:07:49.590286 4928 raft_consensus.cc:2354] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:49.591408 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:49.378123 7842 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 0.794s user 0.003s sys 0.006s
I20260806 04:07:49.546082 7643 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:49.538789 4808 rpcz_store.cc:276] 0806 04:07:48.402637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.063884 (+661247us) service_pool.cc:227] Handling call
0806 04:07:49.064218 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:49.517622 (+453404us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.517835 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:49.519996 5859 pending_rounds.cc:76] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Aborting all ops after (but not including) 5
W20260806 04:07:49.294914 7308 consensus_meta.cc:287] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.062s user 0.017s sys 0.000s
I20260806 04:07:49.544930 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.540279 5038 rpcz_store.cc:276] 0806 04:07:48.514107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.947955 (+433848us) service_pool.cc:227] Handling call
0806 04:07:49.536427 (+588472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:49.038683 7833 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:49.546528 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:49.548517 4876 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.632s (SENT)
I20260806 04:07:49.606745 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.607036 4927 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:49.548708 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:49.333451 7786 log.cc:924] Time spent T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 2.017s user 0.003s sys 0.000s
I20260806 04:07:49.551160 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:49.551386 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:49.420490 7603 raft_consensus.cc:3032] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:49.552371 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.380750 7691 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:49.584108 4804 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:49.377345 7848 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:49.591521 7857 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.392702 7821 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:49.020437 7816 consensus_queue.cc:1046] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.519s
W20260806 04:07:49.503628 7860 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260806 04:07:49.073695 7674 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:49.550468 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:49.646413 5034 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:49.533116 7737 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:49.582887 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.531585 7743 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:49.583321 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.655192 5862 raft_consensus.cc:2396] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:49.496223 7690 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:49.520330 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 491986us
I20260806 04:07:49.661984 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:49.646570 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1962) took 2923 ms. Trace:
I20260806 04:07:49.584017 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:49.582973 4814 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:49.538466 4810 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.059s user 0.004s sys 0.011s
I20260806 04:07:49.600493 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:49.582479 4919 raft_consensus.cc:2379] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:49.215497 7759 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.367s
I20260806 04:07:49.704348 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:49.538370 7852 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:07:49.607414 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 58929us
W20260806 04:07:49.709872 4876 leader_election.cc:334] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.649s (SENT)
I20260806 04:07:49.710108 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:49.710294 4876 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:49.710808 5034 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:49.533859 7836 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:49.713075 4876 leader_election.cc:334] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.564s (SENT)
I20260806 04:07:49.608196 4816 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:49.597719 5856 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:49.606015 4931 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:49.597731 7371 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260806 04:07:49.605454 5859 pending_rounds.cc:97] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 04:07:49.610492 7833 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:49.611146 7308 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:49.611375 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:49.770015 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:49.770840 5038 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:49.770908 5034 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:49.592690 7711 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.612039 7594 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:49.666731 7723 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:49.583856 7678 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.615604 5026 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:49.715988 7714 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.403s user 0.009s sys 0.011s
W20260806 04:07:49.599525 7731 consensus_meta.cc:287] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.145s user 0.014s sys 0.000s
I20260806 04:07:49.607510 4923 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:07:49.619691 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 80658.8us
I20260806 04:07:49.826469 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:49.827620 5026 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:49.532580 7829 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:49.620762 5037 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:49.618582 4920 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:49.127388 7511 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:49.748054 6970 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.389176 7671 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:49.629671 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:49.545511 7762 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.603619 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.638947 7655 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.638886 7857 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:49.660347 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:49.911295 4817 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:49.912204 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:49.912382 4882 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:49.913033 4817 raft_consensus.cc:2396] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:49.660367 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.665848 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 283) took 1888 ms. Trace:
I20260806 04:07:49.532095 7830 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.617733 7643 raft_consensus.cc:686] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:49.662983 4805 rpcz_store.cc:276] 0806 04:07:46.722815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.726925 (+ 4110us) service_pool.cc:227] Handling call
0806 04:07:46.727366 (+ 441us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.757688 (+ 30322us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:46.757719 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:46.771989 (+ 14270us) log.cc:842] Serialized 26 byte log entry
0806 04:07:46.908443 (+136454us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:46.908495 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:46.908571 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:49.331838 (+2423267us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b24002716e8. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d688acf4 00007f23d6888dfb 00007f23d6917559 00007f23d69171e5 00007f23d691682d 00007f23d95c546e 00007f23d95dcc0c 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551
0806 04:07:49.620560 (+288722us) raft_consensus.cc:1671] finished
0806 04:07:49.620582 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:49.646470 (+ 25888us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37877,"raft.queue_time_us":8327,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":8051,"spinlock_wait_cycles":62998272,"thread_start_us":57400,"threads_started":1,"wal-append.queue_time_us":215054}
I20260806 04:07:49.618391 7766 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:49.618206 7603 raft_consensus.cc:729] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:49.674389 4808 raft_consensus.cc:2379] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:49.697688 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1593) took 1368 ms. Trace:
I20260806 04:07:49.663830 4809 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:49.701602 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1974) took 2971 ms. Trace:
I20260806 04:07:49.958158 4818 rpcz_store.cc:276] 0806 04:07:46.740284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.838438 (+ 98154us) service_pool.cc:227] Handling call
0806 04:07:46.870880 (+ 32442us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:46.885982 (+ 15102us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.701193 (+2815211us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b54000df054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:49.701495 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8372096}
I20260806 04:07:49.627921 7691 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.713665 4876 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:49.841842 7731 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:49.612428 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 287) took 3304 ms. Trace:
I20260806 04:07:49.967896 4924 rpcz_store.cc:276] 0806 04:07:46.307640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.307844 (+ 204us) service_pool.cc:227] Handling call
0806 04:07:46.312453 (+ 4609us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:47.365414 (+1052961us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.612140 (+2246726us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001b85d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:49.612336 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2419723648}
I20260806 04:07:49.665517 7737 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:49.791688 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:49.791042 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.539841 7609 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:07:49.713701 7864 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.286s user 0.006s sys 0.002s
I20260806 04:07:49.828635 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.709062 7816 consensus_queue.cc:1225] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer 22fe557f852d4415a96fd683f970cc24 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260806 04:07:49.791296 7860 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260806 04:07:49.840498 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:49.987649 4927 raft_consensus.cc:1261] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:49.532317 7845 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.988080 5863 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:49.748016 7730 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:49.769248 5859 raft_consensus.cc:2866] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:49.769439 7833 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:49.666518 7674 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:49.824245 7714 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.537s user 0.010s sys 0.012s
I20260806 04:07:49.750567 7752 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:49.851342 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.011320 5026 raft_consensus.cc:2379] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
W20260806 04:07:49.836134 7854 log.cc:924] Time spent T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.410s user 0.001s sys 0.000s
I20260806 04:07:49.898423 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:49.845137 7876 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.967375 7739 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:49.842170 7884 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:49.700034 7850 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.147s
I20260806 04:07:49.913879 4810 rpcz_store.cc:276] 0806 04:07:47.778579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:47.805191 (+ 26612us) service_pool.cc:227] Handling call
0806 04:07:47.884323 (+ 79132us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:47.884677 (+ 354us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:07:48.479116 (+594439us) log.cc:842] Serialized 14 byte log entry
0806 04:07:49.665371 (+1186255us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:49.665423 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:49.665470 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.665606 (+ 136us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:49.665768 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":602,"prepare.queue_time_us":176555,"prepare.run_cpu_time_us":7388,"prepare.run_wall_time_us":62320,"raft.queue_time_us":36662,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":116706,"thread_start_us":1752672,"threads_started":2,"wal-append.queue_time_us":1158153}
I20260806 04:07:49.898136 7655 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:49.919883 7870 log.cc:924] Time spent T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:07:49.920468 7836 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:49.797197 7308 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:49.910082 5034 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:49.829602 7678 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.858889 7511 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:49.957278 5036 rpcz_store.cc:276] 0806 04:07:48.329251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.500328 (+171077us) service_pool.cc:227] Handling call
0806 04:07:48.506173 (+ 5845us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:48.706928 (+200755us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:48.706959 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:48.764840 (+ 57881us) log.cc:842] Serialized 26 byte log entry
0806 04:07:49.120486 (+355646us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:49.463099 (+342613us) log.cc:842] Serialized 14 byte log entry
0806 04:07:49.463431 (+ 332us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.639789 (+176358us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540008d254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:49.639845 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:49.650457 (+ 10612us) raft_consensus.cc:1671] finished
0806 04:07:49.696091 (+ 45634us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:49.697606 (+ 1515us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13363,"prepare.queue_time_us":386199,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":161687,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":7760,"spinlock_wait_cycles":429866880,"thread_start_us":536512,"threads_started":2,"wal-append.queue_time_us":538814}
W20260806 04:07:49.961422 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 248379us
I20260806 04:07:49.975780 7691 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.856459 6970 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:49.977794 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:49.977886 5862 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:49.923162 7759 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.052574 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.052481 7759 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:50.053762 4926 raft_consensus.cc:2379] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:49.958061 7866 log.cc:924] Time spent T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.369s user 0.000s sys 0.001s
I20260806 04:07:49.986177 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1982) took 1568 ms. Trace:
W20260806 04:07:49.976305 7868 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.258s user 0.001s sys 0.000s
I20260806 04:07:50.009840 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1983) took 1591 ms. Trace:
I20260806 04:07:49.957791 7594 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:50.009354 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.013679 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1586) took 1686 ms. Trace:
I20260806 04:07:50.012112 5037 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:50.061285 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.061656 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1874) took 1136 ms. Trace:
I20260806 04:07:49.841117 7875 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:50.015722 4931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:50.068137 4931 raft_consensus.cc:595] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:50.068590 4931 raft_consensus.cc:603] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:50.068068 7875 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.068744 7726 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:50.069397 4919 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:07:50.069761 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.071022 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.071429 7691 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:50.072470 6970 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.073096 7655 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:50.074290 7881 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20260806 04:07:50.075158 7880 log.cc:924] Time spent T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20260806 04:07:50.013810 7730 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.077389 7730 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.060756 5032 rpcz_store.cc:276] 0806 04:07:48.326823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.326976 (+ 153us) service_pool.cc:227] Handling call
0806 04:07:48.327323 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:49.062062 (+734739us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.007655 (+945593us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54000ce254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:50.013574 (+ 5919us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1450177408}
I20260806 04:07:49.976634 7737 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:50.027104 4882 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:49.989822 7845 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:50.034472 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 187709us
W20260806 04:07:50.079867 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 52879.4us
I20260806 04:07:50.040262 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1863) took 1339 ms. Trace:
I20260806 04:07:50.080915 5034 rpcz_store.cc:276] 0806 04:07:48.701184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.748570 (+1047386us) service_pool.cc:227] Handling call
0806 04:07:50.040199 (+291629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.081934 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.041625 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:50.043254 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.083853 5026 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:50.084427 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 188) took 1103 ms. Trace:
I20260806 04:07:50.054867 4818 rpcz_store.cc:276] 0806 04:07:48.418038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.959265 (+1541227us) service_pool.cc:227] Handling call
0806 04:07:49.985614 (+ 26349us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:49.985815 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:49.985838 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:49.985884 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:49.985912 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.985995 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:49.986117 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.062124 5037 rpcz_store.cc:276] 0806 04:07:48.924938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.846376 (+921438us) service_pool.cc:227] Handling call
0806 04:07:50.061580 (+215204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.060381 4808 rpcz_store.cc:276] 0806 04:07:48.418338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.994944 (+1576606us) service_pool.cc:227] Handling call
0806 04:07:50.009309 (+ 14365us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.009520 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.009546 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.009587 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:50.009614 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.009665 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.009778 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:49.772068 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.040277 7371 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.055709 7603 consensus_queue.cc:260] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.088755 5034 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.089284 4813 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.089345 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1880) took 1050 ms. Trace:
I20260806 04:07:50.090063 5034 rpcz_store.cc:276] 0806 04:07:49.061161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.081636 (+1020475us) service_pool.cc:227] Handling call
0806 04:07:50.089279 (+ 7643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.091267 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1991) took 1003 ms. Trace:
I20260806 04:07:50.091727 4813 rpcz_store.cc:276] 0806 04:07:49.087400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.542897 (+455497us) service_pool.cc:227] Handling call
0806 04:07:50.091176 (+548279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.093582 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1993) took 1005 ms. Trace:
I20260806 04:07:50.081475 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1851) took 1757 ms. Trace:
I20260806 04:07:50.100719 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.101447 5036 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.087005 5038 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.084774 5026 rpcz_store.cc:276] 0806 04:07:48.980855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.014874 (+1034019us) service_pool.cc:227] Handling call
0806 04:07:50.084361 (+ 69487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.108254 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.109253 5034 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:50.111202 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.098613 5029 rpcz_store.cc:276] 0806 04:07:48.324215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.324394 (+ 179us) service_pool.cc:227] Handling call
0806 04:07:48.415857 (+ 91463us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:49.529696 (+1113839us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:07:49.529733 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:49.547963 (+ 18230us) log.cc:842] Serialized 26 byte log entry
0806 04:07:49.667091 (+119128us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:49.667127 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:49.667187 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:50.081178 (+413991us) raft_consensus.cc:1671] finished
0806 04:07:50.081207 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.081397 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9210,"raft.queue_time_us":286138,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":57703,"thread_start_us":1168180,"threads_started":2,"wal-append.queue_time_us":185848}
I20260806 04:07:50.114236 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.119709 4882 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:50.119728 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.121224 5038 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:50.121592 5038 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:50.125327 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1865) took 1203 ms. Trace:
I20260806 04:07:50.125718 5037 rpcz_store.cc:276] 0806 04:07:48.921860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.113911 (+1192051us) service_pool.cc:227] Handling call
0806 04:07:50.125264 (+ 11353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.055531 7711 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.078810 7888 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.063474 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:50.152926 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.064689 7594 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:50.135917 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.136938 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.138477 7674 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.140419 7759 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.142438 7308 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.034551 7884 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.143205 7762 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:50.144743 5035 consensus_meta.cc:287] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.621s user 0.008s sys 0.000s
W20260806 04:07:50.145602 7883 log.cc:924] Time spent T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260806 04:07:50.147001 7857 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:50.147749 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:50.120635 7870 log.cc:924] Time spent T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 04:07:50.134471 4818 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:50.343463 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 291) took 1595 ms. Trace:
I20260806 04:07:50.343850 4818 rpcz_store.cc:276] 0806 04:07:48.748417 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.085572 (+1337155us) service_pool.cc:227] Handling call
0806 04:07:50.134289 (+ 48717us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.343177 (+208888us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.343394 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.151829 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.344944 5036 raft_consensus.cc:2362] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:50.345321 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1883) took 1203 ms. Trace:
I20260806 04:07:50.345490 5036 rpcz_store.cc:276] 0806 04:07:49.142119 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.121121 (+979002us) service_pool.cc:227] Handling call
0806 04:07:50.345276 (+224155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.135322 7714 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:50.059993 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 241) took 1090 ms. Trace:
I20260806 04:07:50.094024 4813 rpcz_store.cc:276] 0806 04:07:49.088020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.092558 (+1004538us) service_pool.cc:227] Handling call
0806 04:07:50.092956 (+ 398us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.093178 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.093213 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.093280 (+ 67us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:50.093323 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.093390 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.093515 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:50.215883 4876 leader_election.cc:334] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.568s (SENT)
I20260806 04:07:50.150806 4770 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:50.309319 5038 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:50.154803 5037 raft_consensus.cc:2362] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:50.155424 5026 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.112085 7875 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.032387 7643 consensus_queue.cc:237] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.113229 7730 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:50.027534 7876 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.325376 5034 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.330636 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:50.331815 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.008s sys 0.037s
I20260806 04:07:50.331409 5035 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:50.149133 7829 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:50.116034 7371 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.309890 7594 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:50.326805 7674 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.316601 7888 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.326680 7852 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.395032 7655 consensus_queue.cc:1046] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.372s
I20260806 04:07:50.768004 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.768111 7653 raft_consensus.cc:3032] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:50.768674 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1980) took 2365 ms. Trace:
I20260806 04:07:50.780562 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1890) took 1397 ms. Trace:
I20260806 04:07:50.348719 4804 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:50.435411 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:50.782341 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 566519us
I20260806 04:07:50.797881 5029 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.798497 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1887) took 1538 ms. Trace:
I20260806 04:07:50.798916 5029 rpcz_store.cc:276] 0806 04:07:49.260414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.348568 (+1088154us) service_pool.cc:227] Handling call
0806 04:07:50.798436 (+449868us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:50.471129 7840 log.cc:924] Time spent T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.707s user 0.003s sys 0.016s
I20260806 04:07:50.774919 5862 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:50.781196 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1893) took 1364 ms (client timeout 1657 ms). Trace:
I20260806 04:07:50.393545 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.778205 4805 rpcz_store.cc:276] 0806 04:07:48.402972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.923206 (+1520234us) service_pool.cc:227] Handling call
0806 04:07:50.010214 (+ 87008us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:50.067012 (+ 56798us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.067040 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:50.067503 (+ 463us) log.cc:842] Serialized 26 byte log entry
0806 04:07:50.341261 (+273758us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:50.341320 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.341413 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:50.768467 (+427054us) raft_consensus.cc:1671] finished
0806 04:07:50.768488 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.768615 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18860,"raft.queue_time_us":116588,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":10871,"thread_start_us":306603,"threads_started":2,"wal-append.queue_time_us":531410}
I20260806 04:07:50.804330 4995 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:50.804504 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.804843 4996 leader_election.cc:398] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:07:50.805214 4996 leader_election.cc:401] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:07:50.805379 7511 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:50.805691 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.805940 7511 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:50.806223 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.788472 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1552) took 2488 ms. Trace:
I20260806 04:07:50.807895 4930 rpcz_store.cc:276] 0806 04:07:48.300654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.462546 (+161892us) service_pool.cc:227] Handling call
0806 04:07:50.788237 (+2325691us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54003548d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:50.788399 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":377600256}
I20260806 04:07:50.768436 7888 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.793174 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.645843 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 289) took 2150 ms. Trace:
I20260806 04:07:50.771081 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1897) took 1150 ms. Trace:
I20260806 04:07:50.809439 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:50.773576 5038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1859) took 2259 ms (client timeout 2682 ms). Trace:
I20260806 04:07:50.807932 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1999) took 1639 ms. Trace:
W20260806 04:07:50.796036 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 645320us
I20260806 04:07:50.812175 4805 rpcz_store.cc:276] 0806 04:07:49.168417 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.806535 (+1638118us) service_pool.cc:227] Handling call
0806 04:07:50.807297 (+ 762us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.807567 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.807590 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.807627 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:50.807658 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.807708 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.807876 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.797655 5037 rpcz_store.cc:276] 0806 04:07:49.383724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.135648 (+751924us) service_pool.cc:227] Handling call
0806 04:07:50.780487 (+644839us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:50.814019 5034 rpcz_store.cc:269] 0806 04:07:49.417521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.090821 (+673300us) service_pool.cc:227] Handling call
0806 04:07:50.781141 (+690320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.809680 5026 rpcz_store.cc:276] 0806 04:07:49.621125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.120435 (+499310us) service_pool.cc:227] Handling call
0806 04:07:50.771009 (+650574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.810228 5036 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.817764 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1885) took 1600 ms. Trace:
I20260806 04:07:50.818264 5036 rpcz_store.cc:276] 0806 04:07:49.217285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.345891 (+1128606us) service_pool.cc:227] Handling call
0806 04:07:50.817689 (+471798us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:50.784102 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.961s user 0.009s sys 0.049s
I20260806 04:07:50.801879 4808 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:07:50.806380 7511 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:50.802160 4816 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:50.422123 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:50.810508 4804 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:50.822124 4818 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:50.809562 4817 rpcz_store.cc:276] 0806 04:07:48.495498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.066684 (+1571186us) service_pool.cc:227] Handling call
0806 04:07:50.066980 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.067157 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:07:50.309213 (+242056us) log.cc:842] Serialized 14 byte log entry
0806 04:07:50.644948 (+335735us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.645088 (+ 140us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:50.645167 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.645358 (+ 191us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.645653 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72424,"prepare.queue_time_us":355031,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":75196,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":84883,"thread_start_us":350566,"threads_started":2}
I20260806 04:07:50.825557 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2005) took 1190 ms. Trace:
W20260806 04:07:50.811703 5038 rpcz_store.cc:269] 0806 04:07:48.513753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.119484 (+1605731us) service_pool.cc:227] Handling call
0806 04:07:50.773508 (+654024us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.827406 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.826004 4817 rpcz_store.cc:276] 0806 04:07:49.635302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.824599 (+1189297us) service_pool.cc:227] Handling call
0806 04:07:50.824954 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.825160 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.825194 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.825245 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:50.825291 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.825365 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.825487 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.831282 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 295) took 1127 ms. Trace:
I20260806 04:07:50.831746 4817 rpcz_store.cc:276] 0806 04:07:49.704354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.830316 (+1125962us) service_pool.cc:227] Handling call
0806 04:07:50.830687 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.830893 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.830927 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.830984 (+ 57us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:50.831026 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.831089 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.831214 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.766546 4928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:50.468637 7896 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.833837 4928 raft_consensus.cc:595] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.834301 4928 raft_consensus.cc:603] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 04:07:50.835295 4876 leader_election.cc:334] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.654s (SENT)
I20260806 04:07:50.835903 4876 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.838574 7762 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
W20260806 04:07:50.839111 7671 consensus_meta.cc:287] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.982s user 0.012s sys 0.000s
I20260806 04:07:50.839728 4931 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:50.840265 7669 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.840780 7669 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:50.453274 5853 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 235) took 4191 ms (client timeout 2090 ms). Trace:
I20260806 04:07:50.777868 7653 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:07:50.842518 5853 rpcz_store.cc:269] 0806 04:07:46.259623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:46.262978 (+ 3355us) service_pool.cc:227] Handling call
0806 04:07:46.530230 (+267252us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:50.422481 (+3892251us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540059ecd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:50.451410 (+ 28929us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":478753920}
I20260806 04:07:50.790970 7730 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:50.844316 7730 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:50.843871 7653 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:50.845384 7653 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:50.774204 5859 rpcz_store.cc:276] 0806 04:07:48.969231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.974789 (+ 5558us) service_pool.cc:227] Handling call
0806 04:07:48.975216 (+ 427us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.034759 (+1059543us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.055770 (+ 21011us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:50.852617 7737 consensus_meta.cc:287] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.725s user 0.005s sys 0.004s
I20260806 04:07:50.853814 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:50.856658 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 50502.6us
I20260806 04:07:50.856974 7845 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:50.857769 4996 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.855779 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.858834 7767 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:50.468099 7884 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.793735 7714 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:50.863179 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.864835 5026 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:50.864737 7714 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:50.867113 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1981) took 2449 ms. Trace:
I20260806 04:07:50.867515 4809 rpcz_store.cc:276] 0806 04:07:48.417685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.958449 (+1540764us) service_pool.cc:227] Handling call
0806 04:07:49.958908 (+ 459us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:50.310188 (+351280us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.310241 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:50.335130 (+ 24889us) log.cc:842] Serialized 26 byte log entry
0806 04:07:50.454702 (+119572us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:50.454752 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.454842 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:50.866870 (+412028us) raft_consensus.cc:1671] finished
0806 04:07:50.866890 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.867048 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28215,"raft.queue_time_us":143053,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":14020,"thread_start_us":329920,"threads_started":2,"wal-append.queue_time_us":413021}
I20260806 04:07:50.870857 7896 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:50.871388 7655 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:07:50.872988 7896 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:50.877177 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 2.52064e+06us
I20260806 04:07:50.828182 5038 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:50.878361 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1898) took 1216 ms. Trace:
I20260806 04:07:50.879142 5038 rpcz_store.cc:276] 0806 04:07:49.661531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.827071 (+1165540us) service_pool.cc:227] Handling call
0806 04:07:50.878223 (+ 51152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.853298 7737 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:50.881945 7833 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.865s user 0.011s sys 0.000s
I20260806 04:07:50.876529 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.882602 7737 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.882387 7833 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:50.883241 5036 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:50.884562 7833 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:50.885114 7511 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:50.885725 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:50.886469 5859 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:50.887907 4996 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.890061 7876 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:50.784040 7830 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:50.890937 7830 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:50.448467 7731 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:50.884295 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 190) took 1902 ms. Trace:
I20260806 04:07:50.895215 5036 rpcz_store.cc:276] 0806 04:07:48.981440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.875861 (+1894421us) service_pool.cc:227] Handling call
0806 04:07:50.884174 (+ 8313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.896428 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.863377 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.898131 4996 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.914884 4996 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:50.916450 4996 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.898236 5859 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:50.900434 4809 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:50.766543 7852 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.881193 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:50.910398 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 92908.9us
I20260806 04:07:50.801918 7875 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:50.936106 5038 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:50.918365 5036 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:50.920959 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 107056us
I20260806 04:07:50.791836 7674 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.924306 7830 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:50.925534 4996 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.914083 7419 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:50.925993 7767 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:50.894402 7594 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:50.927467 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:07:50.909745 5812 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:50.912446 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 77190.6us
W20260806 04:07:50.928460 4879 connection.cc:232] RPC call timeout handler was delayed by -1.6103555341251194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
I20260806 04:07:50.930828 5034 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
W20260806 04:07:50.942507 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1861) took 2428 ms (client timeout 3082 ms). Trace:
W20260806 04:07:50.942886 4876 leader_election.cc:334] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.949s (SENT)
W20260806 04:07:50.943059 5034 rpcz_store.cc:269] 0806 04:07:48.514427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.855255 (+2340828us) service_pool.cc:227] Handling call
0806 04:07:50.942453 (+ 87198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.943437 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:50.943816 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:50.944116 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.944768 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:50.939129 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:50.933384 7661 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.265s user 0.051s sys 0.070s
I20260806 04:07:50.926096 7833 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:50.941627 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 296) took 1233 ms. Trace:
I20260806 04:07:50.959949 5029 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:50.942337 4879 connection.cc:232] RPC call timeout handler was delayed by -1.3520555342547596s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
W20260806 04:07:50.960602 4879 connection.cc:232] RPC call timeout handler was delayed by -1.2984000318683684s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
W20260806 04:07:50.960938 4879 connection.cc:232] RPC call timeout handler was delayed by -1.1127815879881382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.209s for a response
W20260806 04:07:50.955832 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46161.2us
W20260806 04:07:50.955124 7661 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.367s user 0.051s sys 0.070s
W20260806 04:07:50.964716 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.009s sys 0.009s
I20260806 04:07:50.958141 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.962121 4817 rpcz_store.cc:276] 0806 04:07:49.708633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.832866 (+1124233us) service_pool.cc:227] Handling call
0806 04:07:50.833188 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.940976 (+107788us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.941007 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.941054 (+ 47us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:50.941100 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.941170 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:50.941529 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":102647,"raft.queue_time_us":103005}
W20260806 04:07:50.965148 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.009s sys 0.009s
I20260806 04:07:50.957507 7898 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:50.941756 7767 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:50.941396 7762 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:50.968781 5038 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:50.967350 4810 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.004s sys 0.058s
I20260806 04:07:50.971132 4928 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 04:07:50.982759 4810 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.004s sys 0.058s
I20260806 04:07:50.982899 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1867) took 2060 ms. Trace:
I20260806 04:07:50.957779 7652 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:50.959589 7830 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:50.989993 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.994031 4928 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:50.978785 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:50.937057 7875 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:50.995036 7875 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:50.995977 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.990008 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.997804 5034 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:50.998489 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1899) took 1225 ms. Trace:
I20260806 04:07:50.998996 5034 rpcz_store.cc:276] 0806 04:07:49.790905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.977431 (+1186526us) service_pool.cc:227] Handling call
0806 04:07:50.998419 (+ 20988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.983515 5038 rpcz_store.cc:276] 0806 04:07:48.922557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.957854 (+2035297us) service_pool.cc:227] Handling call
0806 04:07:50.982834 (+ 24980us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.975528 4995 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:50.985126 4768 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:50.986433 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:50.964385 7896 raft_consensus.cc:465] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Not starting pre-election -- already a leader
I20260806 04:07:50.966127 7876 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:50.996622 4923 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:50.983911 7767 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:50.980134 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:50.970310 5026 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:51.002774 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:50.986613 7556 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:50.955821 7730 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.027282 4817 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:51.028292 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1896) took 1411 ms. Trace:
I20260806 04:07:51.064651 5026 rpcz_store.cc:276] 0806 04:07:49.617229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.862341 (+1245112us) service_pool.cc:227] Handling call
0806 04:07:51.028203 (+165862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:50.966439 7754 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 04:07:51.052481 5034 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:51.020938 5036 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:51.069460 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1900) took 1220 ms. Trace:
I20260806 04:07:51.069525 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:51.069969 5036 rpcz_store.cc:276] 0806 04:07:49.848887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.896112 (+1047225us) service_pool.cc:227] Handling call
0806 04:07:51.069374 (+173262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:51.071200 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:51.004673 7731 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:51.072114 5863 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:07:51.006206 6970 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.040s sys 0.037s
W20260806 04:07:51.070596 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 67375.1us
I20260806 04:07:51.076822 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.072075 5036 raft_consensus.cc:2362] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5. Re-sending same reply.
I20260806 04:07:51.055542 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:50.989292 7890 consensus_queue.cc:1035] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260806 04:07:50.993621 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:51.079196 5026 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:51.005398 7762 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:51.079138 4768 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:50.982865 7761 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260806 04:07:51.081207 5038 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:51.077745 5034 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5. Re-sending same reply.
I20260806 04:07:51.095045 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1903) took 1069 ms. Trace:
I20260806 04:07:51.095539 5034 rpcz_store.cc:276] 0806 04:07:50.025925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.000217 (+974292us) service_pool.cc:227] Handling call
0806 04:07:51.094962 (+ 94745us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:51.097609 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.098408 5863 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:51.100062 4817 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:51.101598 5816 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:51.105602 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.106312 5036 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:51.106642 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:51.108404 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:51.109848 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:51.005168 4919 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:51.111542 5038 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 04:07:51.112322 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1594) took 2782 ms. Trace:
I20260806 04:07:51.112671 5038 rpcz_store.cc:276] 0806 04:07:48.329580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.106529 (+2776949us) service_pool.cc:227] Handling call
0806 04:07:51.106934 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:51.112015 (+ 5081us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.112270 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:51.113384 5867 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:07:51.110999 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34040 (request call id 641) took 2814 ms. Trace:
W20260806 04:07:51.088161 6970 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s user 0.040s sys 0.037s
I20260806 04:07:51.114847 5030 rpcz_store.cc:276] 0806 04:07:48.305060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.305211 (+ 151us) service_pool.cc:227] Handling call
0806 04:07:51.060920 (+2755709us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b540032fad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23de89fc6e 00007f23d7f96807 00007f23d7f96792 00007f23d7f966c1 00007f23d7f9663c 00007f23d7f958f2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:51.098921 (+ 38001us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154999040}
I20260806 04:07:51.117282 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:51.115731 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 151409us
I20260806 04:07:50.987731 7898 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.118536 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2003) took 1572 ms. Trace:
I20260806 04:07:51.004925 7888 consensus_queue.cc:1035] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260806 04:07:51.120085 5036 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:51.141707 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1596) took 2811 ms. Trace:
I20260806 04:07:51.142096 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.142194 5036 rpcz_store.cc:276] 0806 04:07:48.330211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.106956 (+2776745us) service_pool.cc:227] Handling call
0806 04:07:51.119778 (+ 12822us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:51.141336 (+ 21558us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.141622 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:51.061780 7876 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:51.120488 5034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02"
dest_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:34050
I20260806 04:07:51.113224 7731 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:51.152668 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.119342 7829 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:51.120602 7586 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:51.121315 4879 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:51.164755 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 43527.9us
I20260806 04:07:51.146188 5034 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:07:51.165800 5034 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:51.137071 5038 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:51.111864 7762 raft_consensus.cc:686] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:51.116356 7912 raft_consensus.cc:2781] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:51.137930 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 198900us
I20260806 04:07:51.169767 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2004) took 1614 ms. Trace:
I20260806 04:07:51.168444 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1597) took 2838 ms. Trace:
I20260806 04:07:51.170951 5038 rpcz_store.cc:276] 0806 04:07:48.330509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.113665 (+2783156us) service_pool.cc:227] Handling call
0806 04:07:51.114025 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:51.168061 (+ 54036us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.168365 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:51.172122 4879 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:51.182929 5034 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.188323 5030 raft_consensus.cc:1226] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:51.186422 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.170173 4818 rpcz_store.cc:276] 0806 04:07:49.555200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.823568 (+1268368us) service_pool.cc:227] Handling call
0806 04:07:50.823947 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.824236 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:07:50.930928 (+106692us) log.cc:842] Serialized 14 byte log entry
0806 04:07:51.169431 (+238503us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:51.169489 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:51.169535 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.169604 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.169713 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98756,"prepare.queue_time_us":106260,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"thread_start_us":221207,"threads_started":1}
I20260806 04:07:51.154340 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.182029 7714 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:51.182262 7912 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:51.202394 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.202728 7912 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:51.203315 4919 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:51.204311 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.205542 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.208657 7898 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:51.209525 7674 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:51.211418 7730 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:51.214064 7914 log.cc:924] Time spent T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260806 04:07:51.198148 7594 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:51.149776 7731 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:51.215269 7731 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:51.215848 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1988) took 2796 ms. Trace:
I20260806 04:07:51.216332 4810 rpcz_store.cc:276] 0806 04:07:48.419671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.026453 (+1606782us) service_pool.cc:227] Handling call
0806 04:07:50.041987 (+ 15534us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:50.128086 (+ 86099us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.128113 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:50.155213 (+ 27100us) log.cc:842] Serialized 26 byte log entry
0806 04:07:50.983197 (+827984us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:50.983235 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:50.983299 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:51.215575 (+232276us) raft_consensus.cc:1671] finished
0806 04:07:51.215602 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.215779 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":118070,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":11198,"thread_start_us":677599,"threads_started":2,"wal-append.queue_time_us":747207}
I20260806 04:07:51.198756 7908 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:51.219089 7908 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:51.179303 7852 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:51.219452 7762 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.219431 7908 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:51.223433 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.225054 7876 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:51.225837 7909 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.237s user 0.001s sys 0.000s
I20260806 04:07:51.226631 7669 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:51.229758 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.198375 7643 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260806 04:07:51.237003 7875 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.239435 4810 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:51.239789 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1553) took 1390 ms. Trace:
I20260806 04:07:51.240767 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.240216 4920 rpcz_store.cc:276] 0806 04:07:49.849551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.850304 (+ 753us) service_pool.cc:227] Handling call
0806 04:07:49.857452 (+ 7148us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.035076 (+177624us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.035112 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.035157 (+ 45us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:50.035188 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.239506 (+1204318us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54000131d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:51.239559 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.239694 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407961472}
I20260806 04:07:51.243175 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 580) took 1533 ms. Trace:
I20260806 04:07:51.243793 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.243633 5857 rpcz_store.cc:276] 0806 04:07:49.709394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.715833 (+ 6439us) service_pool.cc:227] Handling call
0806 04:07:49.859052 (+143219us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.115702 (+256650us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.119046 (+ 3344us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.131319 (+ 12273us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:50.134668 (+ 3349us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.242912 (+1108244us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54006905d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:51.242972 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.243110 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":379177344}
I20260806 04:07:51.245422 4920 raft_consensus.cc:2396] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:07:51.245927 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.247062 4919 raft_consensus.cc:2396] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:51.247238 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1559) took 1284 ms. Trace:
I20260806 04:07:51.247586 4925 rpcz_store.cc:276] 0806 04:07:49.962420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.962723 (+ 303us) service_pool.cc:227] Handling call
0806 04:07:49.963138 (+ 415us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.017847 (+ 54709us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.017892 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:50.017952 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:50.018002 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.247018 (+1229016us) spinlock_profiling.cc:243] Waited 54.4 ms on lock 0x7b54000f6c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:51.247064 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.247185 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125165312}
W20260806 04:07:51.249329 7846 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 2.582s user 0.016s sys 0.028s
I20260806 04:07:51.252143 7671 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:51.253297 7875 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:51.199533 7829 raft_consensus.cc:686] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:51.254773 7829 consensus_queue.cc:237] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:51.255785 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1592) took 2926 ms (client timeout 2101 ms). Trace:
W20260806 04:07:51.258112 5028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1868) took 2335 ms (client timeout 1654 ms). Trace:
W20260806 04:07:51.258618 5028 rpcz_store.cc:269] 0806 04:07:48.923062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.257844 (+2334782us) service_pool.cc:214] Skipping call since client already timed out
0806 04:07:51.258014 (+ 170us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 04:07:51.259135 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 316282us
I20260806 04:07:51.252024 7908 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:51.256336 5035 rpcz_store.cc:269] 0806 04:07:48.328934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.471498 (+142564us) service_pool.cc:227] Handling call
0806 04:07:51.255535 (+2784037us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54002d1ed4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:51.255665 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1136757504}
I20260806 04:07:51.213735 7511 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:51.278914 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 342098us
I20260806 04:07:51.242360 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.282529 5034 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:51.286792 4992 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:51.305510 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.305958 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:34050 (request call id 1595) took 2953 ms. Trace:
I20260806 04:07:51.381258 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:51.437081 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 303572us
W20260806 04:07:51.439247 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 138051us
I20260806 04:07:51.408563 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:51.429332 5037 raft_consensus.cc:2354] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5.
I20260806 04:07:51.281672 7830 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:51.282336 7655 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260806 04:07:51.265982 7852 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:51.440052 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:51.458521 5863 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:51.307894 5034 rpcz_store.cc:276] 0806 04:07:48.329894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.107046 (+2777152us) service_pool.cc:227] Handling call
0806 04:07:51.283137 (+176091us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:51.441045 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.404565 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.463474 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.468412 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.469481 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.458009 7896 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:51.515985 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.452742 5857 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:51.539148 5034 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:51.539470 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2010) took 1192 ms. Trace:
I20260806 04:07:51.549816 4818 rpcz_store.cc:276] 0806 04:07:50.346986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.198642 (+851656us) service_pool.cc:227] Handling call
0806 04:07:51.198949 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:51.199197 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:07:51.283934 (+ 84737us) log.cc:842] Serialized 14 byte log entry
0806 04:07:51.539093 (+255159us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:51.539152 (+ 59us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:51.539202 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.539268 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.539413 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55853,"prepare.queue_time_us":100771,"prepare.run_cpu_time_us":6399,"prepare.run_wall_time_us":93836,"raft.queue_time_us":55946,"thread_start_us":198787,"threads_started":1}
I20260806 04:07:51.551515 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.554862 5863 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:51.556514 4818 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:51.557335 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.558343 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.558938 4818 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:51.558941 4770 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:51.559631 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:51.548938 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 390896us
I20260806 04:07:51.560660 4810 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:51.562336 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 121571us
W20260806 04:07:51.566776 4876 leader_election.cc:334] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.959s (SENT)
I20260806 04:07:51.567396 4876 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:51.457545 7830 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.467561 7852 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:51.569514 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.761709 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.782473 4818 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:51.783870 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:51.785713 4768 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:51.683284 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1561) took 1705 ms. Trace:
I20260806 04:07:51.582219 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1606) took 3245 ms. Trace:
I20260806 04:07:51.785854 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1602) took 3450 ms. Trace:
W20260806 04:07:51.800706 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 233961us
I20260806 04:07:51.597810 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:51.803706 5034 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:51.797977 4926 rpcz_store.cc:276] 0806 04:07:49.978132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.322971 (+344839us) service_pool.cc:227] Handling call
0806 04:07:50.349333 (+ 26362us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:50.784695 (+435362us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.784732 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:50.785321 (+ 589us) log.cc:842] Serialized 26 byte log entry
0806 04:07:51.266339 (+481018us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:51.266374 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.266444 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:51.683026 (+416582us) raft_consensus.cc:1671] finished
0806 04:07:51.683049 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.683220 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":515653,"raft.queue_time_us":429399,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":14239,"thread_start_us":423354,"threads_started":1,"wal-append.queue_time_us":845610}
I20260806 04:07:51.807523 4930 raft_consensus.cc:1226] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:51.799428 5035 rpcz_store.cc:276] 0806 04:07:48.336834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.284598 (+2947764us) service_pool.cc:227] Handling call
0806 04:07:51.284984 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:51.581753 (+296769us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:51.581797 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:51.581855 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:51.581909 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.581975 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.582143 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20825,"raft.queue_time_us":185726,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":15484,"thread_start_us":296251,"threads_started":1}
I20260806 04:07:51.616008 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:51.761838 7920 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260806 04:07:51.809760 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:51.801139 5036 rpcz_store.cc:276] 0806 04:07:48.335618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.143428 (+2807810us) service_pool.cc:227] Handling call
0806 04:07:51.143941 (+ 513us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:51.144913 (+ 972us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:51.144955 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:51.161639 (+ 16684us) log.cc:842] Serialized 26 byte log entry
0806 04:07:51.433713 (+272074us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:51.433761 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.433841 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:51.672935 (+239094us) raft_consensus.cc:1671] finished
0806 04:07:51.672963 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.785790 (+112827us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3497,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":14852,"thread_start_us":226737,"threads_started":1,"wal-append.queue_time_us":488938}
I20260806 04:07:51.542526 7898 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:51.802933 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:51.810956 5863 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:51.832217 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:51.833107 4818 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:51.820899 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:51.835196 5036 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:07:51.836676 4876 leader_election.cc:334] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.197s (SENT)
I20260806 04:07:51.837607 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:51.573014 7876 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:51.542557 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.852542 5863 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:51.592515 7927 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:51.618026 7511 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:51.863586 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.621832 7671 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:51.577059 6970 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:51.818753 7930 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.335s user 0.001s sys 0.000s
I20260806 04:07:51.845186 7903 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:51.870893 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:51.873318 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 66264.4us
W20260806 04:07:51.874274 5812 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.897s (SENT)
I20260806 04:07:51.874722 5034 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:51.830811 5867 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5. Re-sending same reply.
I20260806 04:07:51.876004 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.857367 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.640954 7908 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:51.817198 7875 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:51.863252 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.877624 5035 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:51.865756 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.851539 7714 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:51.862099 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.878535 4925 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:51.882624 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.883764 4926 raft_consensus.cc:2396] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:07:51.876781 5034 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:51.889146 5036 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
W20260806 04:07:51.864912 7767 consensus_meta.cc:287] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.798s user 0.010s sys 0.007s
W20260806 04:07:51.867319 7912 consensus_meta.cc:287] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.664s user 0.013s sys 0.002s
I20260806 04:07:51.890460 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.829453 7946 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260806 04:07:51.862442 7957 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:51.905860 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.849776 7898 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:51.870481 7511 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:51.935009 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.863241 5853 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:51.940989 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:51.951093 4926 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:51.954497 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.925529 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:51.971789 5034 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:51.941519 4995 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:51.902055 5036 raft_consensus.cc:2396] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 04:07:51.965442 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.958254 7890 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:51.891440 7876 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:51.901331 7958 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:51.936589 7925 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.356s user 0.000s sys 0.001s
I20260806 04:07:51.974217 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:52.041085 5034 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:51.937913 5035 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:52.033109 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:52.042981 4925 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:51.928278 7924 log.cc:924] Time spent T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
I20260806 04:07:52.033627 4930 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
W20260806 04:07:51.929339 7954 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
W20260806 04:07:51.928048 7922 log.cc:924] Time spent T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260806 04:07:52.040205 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2002) took 2498 ms. Trace:
I20260806 04:07:52.049817 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:51.973472 7731 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.758s user 0.015s sys 0.005s
I20260806 04:07:51.879993 6970 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:51.930789 7830 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:52.053707 4879 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:52.053097 4805 rpcz_store.cc:276] 0806 04:07:49.542300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.813589 (+1271289us) service_pool.cc:227] Handling call
0806 04:07:50.814016 (+ 427us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:50.814306 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:50.814341 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:50.904023 (+ 89682us) log.cc:842] Serialized 26 byte log entry
0806 04:07:51.111889 (+207866us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:51.111927 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.111988 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:52.039973 (+927985us) raft_consensus.cc:1671] finished
0806 04:07:52.039995 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.040132 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":169801,"threads_started":1,"wal-append.queue_time_us":407438}
I20260806 04:07:52.065328 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:52.065953 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.066087 4879 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:51.968417 7957 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:52.058584 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1607) took 3721 ms. Trace:
I20260806 04:07:52.067813 5036 rpcz_store.cc:276] 0806 04:07:48.337118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.057579 (+3720461us) service_pool.cc:227] Handling call
0806 04:07:52.057963 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.058181 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.058219 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.058270 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:52.058311 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.058377 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.058518 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.066619 4805 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:52.060000 7720 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:52.071293 7720 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.071542 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:52.072907 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.073608 4882 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:52.073989 7720 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.073989 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.075273 7936 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:52.075950 7936 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:07:52.076308 7949 log.cc:924] Time spent T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260806 04:07:52.076309 7936 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:52.071811 7932 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:52.077297 7908 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:07:52.078238 7962 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20260806 04:07:52.078616 4879 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:52.079330 7970 log.cc:924] Time spent T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20260806 04:07:52.079883 7932 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:52.080333 7932 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:52.080742 7932 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:52.080919 7943 log.cc:924] Time spent T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
I20260806 04:07:52.081568 7946 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:52.083021 4879 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:52.070487 4770 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:52.087426 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2006) took 2074 ms. Trace:
I20260806 04:07:52.088919 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.090936 5034 raft_consensus.cc:2396] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:07:51.974555 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:51.865912 7927 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.113063 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1857) took 3600 ms. Trace:
I20260806 04:07:52.113502 5034 rpcz_store.cc:276] 0806 04:07:48.513059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.103899 (+3590840us) service_pool.cc:227] Handling call
0806 04:07:52.112572 (+ 8673us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.112751 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.112785 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.112824 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:52.112853 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.112907 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.113004 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.115648 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.087803 4817 rpcz_store.cc:276] 0806 04:07:50.013293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.112448 (+1099155us) service_pool.cc:227] Handling call
0806 04:07:51.114026 (+ 1578us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:51.147742 (+ 33716us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:51.147785 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:51.148318 (+ 533us) log.cc:842] Serialized 26 byte log entry
0806 04:07:51.449868 (+301550us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:51.449912 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.449998 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:52.087213 (+637215us) raft_consensus.cc:1671] finished
0806 04:07:52.087232 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.087370 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72963,"raft.queue_time_us":43052,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":8264,"thread_start_us":242403,"threads_started":1,"wal-append.queue_time_us":552414}
I20260806 04:07:52.119504 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:52.104698 4992 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.003s sys 0.020s
W20260806 04:07:52.120569 4992 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.761s user 0.003s sys 0.021s
W20260806 04:07:52.120980 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 834266us
I20260806 04:07:52.120599 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.121976 4992 leader_election.cc:398] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:52.122331 4992 leader_election.cc:401] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:07:52.123438 4992 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:52.126119 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:51.903991 7767 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.144294 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:52.145395 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1852) took 3749 ms. Trace:
I20260806 04:07:52.147976 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.149986 5812 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:52.143476 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 60613.2us
I20260806 04:07:51.902932 7912 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.167269 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1858) took 3653 ms. Trace:
I20260806 04:07:52.152458 5035 rpcz_store.cc:276] 0806 04:07:48.396156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.059073 (+3662917us) service_pool.cc:227] Handling call
0806 04:07:52.144605 (+ 85532us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.145016 (+ 411us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.145046 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.145091 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:52.145123 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.145178 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.145316 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.153437 4876 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:52.153574 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.169219 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:52.170320 4923 raft_consensus.cc:2379] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:52.155870 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.152738 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:52.154410 4879 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:52.167768 5034 rpcz_store.cc:276] 0806 04:07:48.513428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.119045 (+3605617us) service_pool.cc:227] Handling call
0806 04:07:52.166773 (+ 47728us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.166971 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.166994 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.167031 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:52.167062 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.167109 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.167210 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.060256 7671 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.064411 7829 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:51.968022 7511 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:52.167646 7961 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.381s user 0.000s sys 0.002s
I20260806 04:07:52.209493 5035 raft_consensus.cc:1261] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:52.210458 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1862) took 3695 ms. Trace:
I20260806 04:07:52.210880 5035 rpcz_store.cc:276] 0806 04:07:48.530694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.168342 (+3637648us) service_pool.cc:227] Handling call
0806 04:07:52.208996 (+ 40654us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.210129 (+ 1133us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.210389 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.213621 4810 raft_consensus.cc:2379] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:52.197216 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.083719 7957 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:52.176486 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.082113 7946 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.177613 4817 raft_consensus.cc:2396] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:07:52.082757 7731 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.151904 4919 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:52.051146 7890 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.051819 7958 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.232563 7980 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:52.235909 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2011) took 1299 ms. Trace:
I20260806 04:07:52.247079 4817 rpcz_store.cc:276] 0806 04:07:50.936059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.224360 (+1288301us) service_pool.cc:227] Handling call
0806 04:07:52.224748 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.235341 (+ 10593us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.235425 (+ 84us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.235469 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:52.235496 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.235570 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.235732 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.205273 7737 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:52.204768 7952 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
I20260806 04:07:52.200543 7852 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:52.255715 4804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 2000) took 3061 ms (client timeout 2046 ms). Trace:
I20260806 04:07:52.244167 4818 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:52.258769 4804 rpcz_store.cc:269] 0806 04:07:49.194333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.629314 (+434981us) service_pool.cc:227] Handling call
0806 04:07:52.255467 (+2626153us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b540011e754. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:52.255599 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1154784512}
I20260806 04:07:52.260100 4818 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 04:07:52.215554 7967 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:52.249943 4817 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:07:52.262161 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 304) took 1193 ms. Trace:
I20260806 04:07:52.250835 7737 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:52.262943 7852 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:52.263440 7737 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:52.263887 7852 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:52.264499 7852 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:52.265077 7852 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:52.265586 7965 log.cc:924] Time spent T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260806 04:07:52.267323 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 1989) took 3180 ms. Trace:
I20260806 04:07:52.240337 7701 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:52.215162 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.267783 4808 rpcz_store.cc:276] 0806 04:07:49.086632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.086624 (+999992us) service_pool.cc:227] Handling call
0806 04:07:50.087682 (+ 1058us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.819823 (+732141us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.267014 (+1447191us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b540039b7d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:52.267261 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1641414528}
I20260806 04:07:52.268666 4805 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:52.278805 7843 log.cc:924] Time spent T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 3.365s user 0.016s sys 0.007s
I20260806 04:07:52.279922 4808 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:07:52.250841 7979 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:52.205430 7671 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.288152 7731 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:52.293081 7816 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:52.273412 7967 raft_consensus.cc:686] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:52.260432 4804 raft_consensus.cc:1226] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:52.270293 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 305) took 1078 ms. Trace:
I20260806 04:07:52.212778 7829 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.311146 7946 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:52.311666 5857 consensus_meta.cc:287] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.699s user 0.010s sys 0.006s
I20260806 04:07:52.312175 5857 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:52.313035 7901 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.283598 4808 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:52.269133 4817 rpcz_store.cc:276] 0806 04:07:51.069038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.249163 (+1180125us) service_pool.cc:227] Handling call
0806 04:07:52.249537 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.261903 (+ 12366us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.262106 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.209620 7876 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.281589 7979 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:52.303795 4818 rpcz_store.cc:276] 0806 04:07:51.183830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.259547 (+1075717us) service_pool.cc:227] Handling call
0806 04:07:52.259889 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:52.261426 (+ 1537us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.261652 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.318926 7979 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:52.319847 4818 raft_consensus.cc:1261] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:07:52.256228 6970 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.314031 7901 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.297380 7927 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.305127 7952 consensus_queue.cc:1046] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 04:07:52.305864 7511 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.308149 7890 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:52.323671 7734 raft_consensus.cc:3032] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:07:52.297873 7967 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.324340 7734 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:52.310936 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:52.327798 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.328588 5862 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:52.315855 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1260) took 1119 ms. Trace:
I20260806 04:07:52.330008 4804 rpcz_store.cc:276] 0806 04:07:51.196179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.260009 (+1063830us) service_pool.cc:227] Handling call
0806 04:07:52.260272 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.315618 (+ 55346us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.315802 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.288622 7671 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.314255 7829 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.250025 7980 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:52.334403 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.317965 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:52.340551 4928 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.017s sys 0.025s
W20260806 04:07:52.342849 4928 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s user 0.017s sys 0.025s
I20260806 04:07:52.348538 4770 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:52.328521 7877 raft_consensus.cc:3032] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:52.320907 7898 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.363312 5862 raft_consensus.cc:1226] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:52.343456 7980 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:52.367343 7798 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:52.368610 7767 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.331504 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 526) took 1031 ms. Trace:
I20260806 04:07:52.354637 7829 consensus_queue.cc:1046] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260806 04:07:52.358165 7890 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:52.341660 7952 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 04:07:52.349499 7958 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.347813 7653 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260806 04:07:51.124228 4813 rpcz_store.cc:276] 0806 04:07:49.546489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.822228 (+1275739us) service_pool.cc:227] Handling call
0806 04:07:50.822616 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:50.822877 (+ 261us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:07:50.888261 (+ 65384us) log.cc:842] Serialized 14 byte log entry
0806 04:07:51.118110 (+229849us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:51.118176 (+ 66us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:51.118235 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:51.118330 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:51.118463 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65582,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":3914,"thread_start_us":229500,"threads_started":1}
I20260806 04:07:52.354637 7877 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:52.326049 7734 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.379796 7890 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:52.386785 7890 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:52.389963 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1601) took 4054 ms. Trace:
I20260806 04:07:52.390480 5030 rpcz_store.cc:276] 0806 04:07:48.335320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.135569 (+2800249us) service_pool.cc:227] Handling call
0806 04:07:51.158515 (+ 22946us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:51.188797 (+ 30282us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.389647 (+1200850us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b54000bcf54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:52.389885 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68008832}
W20260806 04:07:52.396756 5037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1888) took 3101 ms (client timeout 2196 ms). Trace:
W20260806 04:07:52.397678 5037 rpcz_store.cc:269] 0806 04:07:49.295618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.853179 (+1557561us) service_pool.cc:227] Handling call
0806 04:07:52.396427 (+1543248us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54000cec54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:52.396653 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1117412608}
I20260806 04:07:52.405838 5037 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:52.379087 5857 rpcz_store.cc:276] 0806 04:07:51.300471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.305721 (+ 5250us) service_pool.cc:227] Handling call
0806 04:07:52.320946 (+1015225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.472457 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.450523 7981 raft_consensus.cc:3032] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:52.475165 7981 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:52.449086 7952 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.478075 7952 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.492775 7609 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:52.493810 7609 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:52.392763 7710 consensus_queue.cc:1046] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
W20260806 04:07:52.474175 5854 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 587) took 2509 ms (client timeout 2146 ms). Trace:
I20260806 04:07:52.507215 7710 raft_consensus.cc:3032] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:52.509837 7710 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:52.504179 7609 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:52.536005 7988 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260806 04:07:52.536775 7737 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.541369 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1872) took 3617 ms. Trace:
I20260806 04:07:52.540710 7936 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.541846 5030 rpcz_store.cc:276] 0806 04:07:48.924321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.391603 (+3467282us) service_pool.cc:227] Handling call
0806 04:07:52.540729 (+149126us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.540981 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.541019 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.541063 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:52.541100 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.541164 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:52.541304 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.543646 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:52.495280 7981 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.546612 7981 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:52.505216 5854 rpcz_store.cc:269] 0806 04:07:49.965041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.978851 (+ 13810us) service_pool.cc:227] Handling call
0806 04:07:52.472196 (+2493345us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:52.474050 (+ 1854us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30155648}
I20260806 04:07:52.557071 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1873) took 3632 ms. Trace:
I20260806 04:07:52.557783 5037 rpcz_store.cc:276] 0806 04:07:48.924625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.402405 (+3477780us) service_pool.cc:227] Handling call
0806 04:07:52.402847 (+ 442us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.418781 (+ 15934us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.556986 (+138205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.544086 7952 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.593803 5030 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:07:52.594739 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1875) took 3669 ms. Trace:
I20260806 04:07:52.595163 5030 rpcz_store.cc:276] 0806 04:07:48.925316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.549112 (+3623796us) service_pool.cc:227] Handling call
0806 04:07:52.593335 (+ 44223us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.594400 (+ 1065us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.594673 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:52.600804 7890 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.604786 7936 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.605857 7936 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.549250 7710 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.609918 7710 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:52.588189 7371 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.613086 7890 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:52.615792 5860 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:52.616544 5860 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 248) took 1638 ms (client timeout 1621 ms). Trace:
W20260806 04:07:52.616969 5860 rpcz_store.cc:269] 0806 04:07:50.977960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.978635 (+ 675us) service_pool.cc:227] Handling call
0806 04:07:52.543448 (+1564813us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:52.616457 (+ 73009us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3871616}
I20260806 04:07:52.613528 7957 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.564635 7734 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:52.606593 7936 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:52.634869 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.022s sys 0.017s
W20260806 04:07:52.635313 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s user 0.022s sys 0.017s
W20260806 04:07:52.646948 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.083s user 0.004s sys 0.045s
W20260806 04:07:52.648999 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.085s user 0.004s sys 0.045s
W20260806 04:07:52.636303 7713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.014s sys 0.026s
W20260806 04:07:52.631390 7674 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.006s user 0.023s sys 0.073s
W20260806 04:07:52.655488 4996 connection.cc:232] RPC call timeout handler was delayed by -1.3596377389039844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20260806 04:07:52.642513 7932 consensus_meta.cc:287] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.561s user 0.003s sys 0.014s
I20260806 04:07:52.658725 7932 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:52.656525 7674 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.034s user 0.024s sys 0.073s
I20260806 04:07:52.671327 7767 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.672381 7767 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:52.654702 7713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.015s sys 0.026s
I20260806 04:07:52.680030 7767 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:52.646616 7609 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.682807 7609 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.687131 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.653275 7936 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:52.690867 5867 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.011s sys 0.013s
W20260806 04:07:52.692656 5867 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.011s sys 0.014s
W20260806 04:07:52.716598 4992 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.006s sys 0.028s
W20260806 04:07:52.782101 4992 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.611s user 0.013s sys 0.038s
W20260806 04:07:52.782887 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 630181us
I20260806 04:07:52.690148 7830 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:52.787326 4809 raft_consensus.cc:2396] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 04:07:52.696503 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.014s sys 0.026s
W20260806 04:07:52.780135 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 70647.4us
I20260806 04:07:52.789700 4992 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:52.798243 4992 connection.cc:232] RPC call timeout handler was delayed by -1.0937425647862256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2404s for a response
I20260806 04:07:52.805509 4925 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:07:52.686295 7932 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:52.810751 4879 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.019s sys 0.024s
W20260806 04:07:52.811549 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.018s sys 0.026s
W20260806 04:07:52.832181 4879 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.019s sys 0.024s
W20260806 04:07:52.853165 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 698797us
I20260806 04:07:52.857293 7980 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.710134 7754 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:52.787858 7884 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260806 04:07:52.752609 7737 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.860255 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:52.865373 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:52.866114 4879 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:52.866257 4930 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:07:52.832599 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 665328us
I20260806 04:07:52.870287 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:52.702821 7942 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.148s
I20260806 04:07:52.875643 4926 raft_consensus.cc:1226] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:52.843010 4882 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:52.863258 7852 consensus_meta.cc:287] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.595s user 0.008s sys 0.011s
I20260806 04:07:52.881387 4813 raft_consensus.cc:1226] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:52.830327 7890 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:52.895179 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:52.872136 7721 raft_consensus.cc:2741] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:07:52.834216 7419 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.849117 7816 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:52.904277 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:52.902666 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:52.899588 7419 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.830549 7587 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:52.911095 7587 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.913200 4925 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:07:52.914470 4931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1429) took 2866 ms (client timeout 2404 ms). Trace:
W20260806 04:07:52.914983 4931 rpcz_store.cc:269] 0806 04:07:50.048137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.070730 (+ 22593us) service_pool.cc:227] Handling call
0806 04:07:52.914245 (+2843515us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54001b80d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:52.914387 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1766263680}
W20260806 04:07:52.905459 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 62612us
W20260806 04:07:52.916440 4814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1257) took 4664 ms (client timeout 1776 ms). Trace:
I20260806 04:07:52.918361 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:52.918581 4814 rpcz_store.cc:269] 0806 04:07:48.251904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:48.439394 (+187490us) service_pool.cc:227] Handling call
0806 04:07:52.916140 (+4476746us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540039bcd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:52.916296 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2627498240}
I20260806 04:07:52.919801 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:52.874210 7908 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:07:52.923336 7957 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:52.945240 7932 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:52.946015 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.946317 7932 raft_consensus.cc:686] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:52.947726 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:52.948249 7932 consensus_queue.cc:237] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.949455 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:52.901482 7890 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:52.951103 7652 raft_consensus.cc:3032] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:52.830436 7830 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.951983 7652 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:52.892558 7661 consensus_queue.cc:1046] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260806 04:07:52.955161 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.956108 7661 raft_consensus.cc:3032] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:52.956789 7661 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:07:52.951411 7979 consensus_meta.cc:287] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.632s user 0.013s sys 0.006s
I20260806 04:07:52.956956 5859 raft_consensus.cc:2354] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5.
I20260806 04:07:52.933480 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:52.961853 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:52.947917 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 593) took 1996 ms. Trace:
I20260806 04:07:52.964494 4930 raft_consensus.cc:2379] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:52.915218 7419 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:52.968777 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.970705 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:52.936794 5038 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:52.973646 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:52.974941 5038 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:07:52.977404 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:52.980012 7830 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:52.967909 5856 rpcz_store.cc:276] 0806 04:07:50.951891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.955285 (+ 3394us) service_pool.cc:227] Handling call
0806 04:07:52.945855 (+1990570us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:52.947798 (+ 1943us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14421888}
I20260806 04:07:52.963704 7661 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.964954 7979 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.991745 7661 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:52.994459 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:52.994844 4770 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:52.995421 5863 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:07:52.996893 4995 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:52.904160 7816 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:52.997776 7884 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:52.972826 7908 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 04:07:52.923417 7743 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:07:52.972276 5859 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 247) took 1995 ms (client timeout 2196 ms). Trace:
I20260806 04:07:52.946317 7980 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:53.003623 4876 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.013s sys 0.076s
I20260806 04:07:52.966305 7652 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.984372 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:52.999161 7830 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:52.925908 7852 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:52.987895 5038 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 6 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260806 04:07:53.007733 4876 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s user 0.013s sys 0.076s
W20260806 04:07:53.022802 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 869354us
I20260806 04:07:53.024622 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:53.007148 7890 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:53.001452 7743 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:52.989256 7419 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260806 04:07:53.026366 7979 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:53.087352 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.095453 5038 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:53.096568 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.097446 5038 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:53.098547 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:53.099332 5038 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:53.010975 4814 raft_consensus.cc:2362] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:53.137495 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:53.138868 7669 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.904s user 0.013s sys 0.021s
W20260806 04:07:53.003054 7888 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.016s sys 0.025s
W20260806 04:07:53.106271 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 81723.5us
I20260806 04:07:53.113358 7890 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:53.146592 7890 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:53.146962 4876 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:53.149458 7890 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:53.011735 7830 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:53.005599 5859 rpcz_store.cc:269] 0806 04:07:50.974987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.986294 (+ 11307us) service_pool.cc:227] Handling call
0806 04:07:52.954844 (+1968550us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:52.970510 (+ 15666us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6136704}
W20260806 04:07:53.093111 7981 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.546s user 0.006s sys 0.009s
I20260806 04:07:53.139776 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:53.000262 7816 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.151588 7830 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:53.151918 5860 raft_consensus.cc:2396] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:53.156009 4814 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:53.141386 7852 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:53.137633 7720 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.009s sys 0.033s
I20260806 04:07:53.142545 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:53.138257 5034 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.015s sys 0.036s
W20260806 04:07:53.193518 5034 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.985s user 0.016s sys 0.036s
W20260806 04:07:53.145978 7669 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.912s user 0.013s sys 0.021s
W20260806 04:07:53.150547 7888 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.879s user 0.025s sys 0.028s
I20260806 04:07:53.133698 7652 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:53.162103 4876 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:53.134219 7995 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:53.159355 7875 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.001s sys 0.055s
W20260806 04:07:53.203034 7875 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.888s user 0.001s sys 0.055s
W20260806 04:07:53.210165 7734 consensus_meta.cc:287] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.582s user 0.005s sys 0.010s
W20260806 04:07:53.200277 5028 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.020s sys 0.025s
W20260806 04:07:53.213325 5028 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.052s user 0.021s sys 0.025s
I20260806 04:07:53.213982 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1603) took 4878 ms. Trace:
I20260806 04:07:53.214313 5028 rpcz_store.cc:276] 0806 04:07:48.335913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.281457 (+2945544us) service_pool.cc:227] Handling call
0806 04:07:51.281850 (+ 393us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:51.305044 (+ 23194us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:51.438731 (+133687us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:51.451025 (+ 12294us) log.cc:842] Serialized 26 byte log entry
0806 04:07:51.851183 (+400158us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:52.148644 (+297461us) log.cc:842] Serialized 14 byte log entry
0806 04:07:53.213681 (+1065037us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.213755 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:53.213795 (+ 40us) raft_consensus.cc:1671] finished
0806 04:07:53.213806 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.213929 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4152,"prepare.queue_time_us":507261,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":243373,"raft.run_cpu_time_us":552,"raft.run_wall_time_us":21599,"thread_start_us":1595858,"threads_started":3,"wal-append.queue_time_us":595716}
I20260806 04:07:53.216161 7609 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:53.216523 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.216846 7609 raft_consensus.cc:686] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:53.217190 5028 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:53.218043 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.217869 7609 consensus_queue.cc:237] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:53.200886 4818 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.012s sys 0.056s
W20260806 04:07:53.195950 7720 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.937s user 0.009s sys 0.033s
I20260806 04:07:53.202162 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:53.218680 5867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 598) took 1453 ms. Trace:
I20260806 04:07:53.218596 5028 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:07:53.195982 8008 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:53.235321 4818 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.893s user 0.015s sys 0.057s
I20260806 04:07:53.236411 4876 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:53.240072 4814 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:53.268803 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:53.253285 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.250155 5038 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:53.277779 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.271790 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1556) took 3356 ms. Trace:
I20260806 04:07:53.272971 4876 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:53.279677 4927 rpcz_store.cc:276] 0806 04:07:49.915028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.915209 (+ 181us) service_pool.cc:227] Handling call
0806 04:07:49.960178 (+ 44969us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:49.988094 (+ 27916us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.271469 (+3283375us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b540006e854. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:53.271712 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58393856}
I20260806 04:07:53.225207 7419 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:53.281508 8002 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:53.282271 8002 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:53.283061 7916 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 2.286s user 0.010s sys 0.000s
I20260806 04:07:53.226064 7995 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:53.257027 7959 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 1.433s user 0.000s sys 0.006s
I20260806 04:07:53.287981 4876 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:53.278631 5038 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:53.235791 5867 rpcz_store.cc:276] 0806 04:07:51.764890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.905065 (+140175us) service_pool.cc:227] Handling call
0806 04:07:51.929139 (+ 24074us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:51.966924 (+ 37785us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:51.966968 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:51.967559 (+ 591us) log.cc:842] Serialized 26 byte log entry
0806 04:07:52.704231 (+736672us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:52.704289 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:52.704386 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:53.218455 (+514069us) raft_consensus.cc:1671] finished
0806 04:07:53.218475 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.218604 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8547,"raft.queue_time_us":37308,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":65725,"thread_start_us":613486,"threads_started":1,"wal-append.queue_time_us":1171031}
I20260806 04:07:53.300370 8003 raft_consensus.cc:2741] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260806 04:07:53.245487 7678 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:53.292687 7419 raft_consensus.cc:686] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:53.304283 7419 consensus_queue.cc:237] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:53.305107 7713 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:53.264719 8008 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:53.307502 8008 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:53.223189 7967 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s user 0.005s sys 0.044s
W20260806 04:07:53.311066 7967 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.006s sys 0.045s
I20260806 04:07:53.309928 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:53.283095 8002 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.313130 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:53.314831 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 302) took 2410 ms. Trace:
I20260806 04:07:53.312397 4814 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:53.314957 4808 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:53.315301 4805 rpcz_store.cc:276] 0806 04:07:50.904399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.218677 (+1314278us) service_pool.cc:227] Handling call
0806 04:07:52.219048 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.277113 (+ 58065us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.314517 (+1037404us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b5400271fd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:53.314763 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99279616}
I20260806 04:07:53.316429 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2021) took 1472 ms. Trace:
I20260806 04:07:53.316588 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:53.316771 4808 rpcz_store.cc:276] 0806 04:07:51.843905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.348608 (+504703us) service_pool.cc:227] Handling call
0806 04:07:53.314750 (+966142us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.315525 (+ 775us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.316260 (+ 735us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b5400271fd0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.316381 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2221122432}
I20260806 04:07:53.318903 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:53.319887 4808 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:53.317830 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:53.324054 4805 raft_consensus.cc:2396] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:07:53.269141 7743 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:53.338682 4817 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.008s sys 0.031s
W20260806 04:07:53.339152 4817 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.011s user 0.008s sys 0.031s
I20260806 04:07:53.347272 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.349112 5038 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:53.350323 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:53.373072 7730 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:53.375393 7912 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:53.377647 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:53.378975 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 307) took 2385 ms. Trace:
I20260806 04:07:53.379179 7671 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:53.381320 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1856) took 4868 ms. Trace:
W20260806 04:07:53.381850 5029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1901) took 3473 ms (client timeout 1565 ms). Trace:
I20260806 04:07:53.381815 5036 rpcz_store.cc:276] 0806 04:07:48.512599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.069312 (+3556713us) service_pool.cc:227] Handling call
0806 04:07:52.069695 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.199425 (+129730us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.199478 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:52.199547 (+ 69us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:52.199606 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.381009 (+1181403us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400320ad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:53.381070 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.381242 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":260703104}
I20260806 04:07:53.383998 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:53.385245 5035 raft_consensus.cc:1226] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:53.386749 4814 raft_consensus.cc:2396] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 04:07:53.382464 5029 rpcz_store.cc:269] 0806 04:07:49.908601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.799671 (+891070us) service_pool.cc:227] Handling call
0806 04:07:53.381600 (+2581929us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b54000bd454. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.381760 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105981824}
I20260806 04:07:53.393105 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.388363 5036 raft_consensus.cc:2362] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4. Re-sending same reply.
I20260806 04:07:53.398331 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.399892 7587 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:53.393978 5038 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:53.409636 4808 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:53.418632 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:53.421202 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:53.422098 5038 raft_consensus.cc:2362] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5. Re-sending same reply.
W20260806 04:07:53.423005 5026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1884) took 4250 ms (client timeout 4605 ms). Trace:
W20260806 04:07:53.423753 5026 rpcz_store.cc:269] 0806 04:07:49.172955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.087904 (+1914949us) service_pool.cc:227] Handling call
0806 04:07:53.422680 (+2334776us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b5400166f54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.422916 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1784308608}
I20260806 04:07:53.425015 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.379408 4928 rpcz_store.cc:276] 0806 04:07:50.993321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.009994 (+ 16673us) service_pool.cc:227] Handling call
0806 04:07:51.010422 (+ 428us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:51.010906 (+ 484us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:07:51.436872 (+425966us) log.cc:842] Serialized 14 byte log entry
0806 04:07:52.344928 (+908056us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:52.345480 (+ 552us) log.cc:842] Serialized 26 byte log entry
0806 04:07:52.353537 (+ 8057us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:52.359626 (+ 6089us) log.cc:842] Serialized 14 byte log entry
0806 04:07:52.699539 (+339913us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b180032e6c0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32638 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:07:53.215862 (+516323us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b48002610d8. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23de8017f1 00007f23de7f88e7 00007f23de7f877a 00007f23de8ec95f 00007f23de8ec627 00007f23de900b2c 00007f23de900ad8 00007f23de900a58 00007f23de900a13 00007f23de8ffcbc 00007f23dd881be6
0806 04:07:53.378509 (+162647us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b54000cd8a0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d98edfb3 00007f23d995af0f 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c
0806 04:07:53.378572 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.378698 (+ 126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:53.378770 (+ 72us) raft_consensus.cc:1671] finished
0806 04:07:53.378781 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.378919 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68905,"prepare.queue_time_us":711695,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":87,"raft.queue_time_us":493652,"raft.run_cpu_time_us":3399,"raft.run_wall_time_us":531398,"spinlock_wait_cycles":141900160,"thread_start_us":1134039,"threads_started":3,"wal-append.queue_time_us":386142,"wal-append.run_cpu_time_us":5014,"wal-append.run_wall_time_us":1521909}
I20260806 04:07:53.331245 8003 consensus_queue.cc:1046] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260806 04:07:53.423427 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.443797 5038 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:53.438894 7816 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:53.445401 5028 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:53.445622 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.446848 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:53.447638 5029 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:53.443702 7830 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:53.452565 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:53.453322 5867 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:53.452610 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:53.459106 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:53.459638 5036 raft_consensus.cc:2379] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 2: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:53.464356 7830 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:53.465659 7995 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:53.467373 5866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 586) took 3603 ms (client timeout 2108 ms). Trace:
I20260806 04:07:53.460100 5038 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:53.470504 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:53.471638 7901 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:53.472601 5859 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:53.472470 7877 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:53.474433 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1870) took 4550 ms. Trace:
I20260806 04:07:53.475101 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:53.467989 5866 rpcz_store.cc:269] 0806 04:07:49.863783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.918826 (+ 55043us) service_pool.cc:227] Handling call
0806 04:07:51.229593 (+1310767us) spinlock_profiling.cc:243] Waited 992 us on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:53.467081 (+2237488us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54005ceed4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.467272 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1440562688}
I20260806 04:07:53.476126 5026 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:53.474864 5035 rpcz_store.cc:276] 0806 04:07:48.923675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.212082 (+3288407us) service_pool.cc:227] Handling call
0806 04:07:53.385039 (+1172957us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.385825 (+ 786us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.474247 (+ 88422us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000bd450. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.474375 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2606574464}
I20260806 04:07:53.477841 5026 raft_consensus.cc:1261] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:53.478518 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1891) took 4084 ms. Trace:
I20260806 04:07:53.479552 5026 rpcz_store.cc:276] 0806 04:07:49.394125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.476971 (+4082846us) service_pool.cc:227] Handling call
0806 04:07:53.477444 (+ 473us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.478248 (+ 804us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.478460 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.471282 7743 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:53.488020 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1904) took 3428 ms. Trace:
I20260806 04:07:53.490484 7603 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:53.485947 7877 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.491858 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:53.487954 7896 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:53.502991 7754 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:53.583092 7754 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.575955 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1866) took 4653 ms. Trace:
I20260806 04:07:53.576192 5026 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
W20260806 04:07:53.582476 7674 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.004s sys 0.037s
I20260806 04:07:53.529680 8002 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:53.538564 5867 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:53.587719 8008 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:53.575636 7556 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.005s sys 0.039s
I20260806 04:07:53.579277 7896 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:53.575565 7816 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:53.567045 7830 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:53.608163 7674 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s user 0.005s sys 0.037s
W20260806 04:07:53.599660 7721 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.011s sys 0.033s
W20260806 04:07:53.634949 7556 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s user 0.005s sys 0.040s
W20260806 04:07:53.643778 7721 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.727s user 0.011s sys 0.034s
I20260806 04:07:53.411794 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 312) took 1212 ms. Trace:
W20260806 04:07:53.701124 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1270) took 2400 ms (client timeout 1706 ms). Trace:
I20260806 04:07:53.702809 4814 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:53.707692 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2012) took 1894 ms. Trace:
W20260806 04:07:53.708104 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1264) took 2479 ms (client timeout 3251 ms). Trace:
I20260806 04:07:53.709268 4804 raft_consensus.cc:1226] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
W20260806 04:07:53.711045 4816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 1990) took 4623 ms (client timeout 2205 ms). Trace:
I20260806 04:07:52.697345 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:07:53.720750 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2023) took 1644 ms. Trace:
W20260806 04:07:53.727075 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.804s user 0.016s sys 0.020s
W20260806 04:07:53.730002 4924 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.007s sys 0.028s
I20260806 04:07:53.575810 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1440) took 1633 ms. Trace:
W20260806 04:07:53.592445 4923 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.009s sys 0.034s
W20260806 04:07:53.730324 4879 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.006s sys 0.015s
W20260806 04:07:53.811391 4879 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.007s sys 0.015s
W20260806 04:07:53.799382 4816 rpcz_store.cc:269] 0806 04:07:49.087088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.731360 (+644272us) service_pool.cc:227] Handling call
0806 04:07:53.710450 (+3979090us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b54002a26d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.710758 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1913964416}
I20260806 04:07:53.811636 7714 raft_consensus.cc:3032] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:53.812278 7714 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:53.813336 4927 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:07:53.813619 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:53.731009 4931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.003s sys 0.025s
I20260806 04:07:53.731693 4995 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:53.730029 4878 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.002s sys 0.019s
I20260806 04:07:53.604852 5034 rpcz_store.cc:276] 0806 04:07:48.922237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.193632 (+3271395us) service_pool.cc:227] Handling call
0806 04:07:52.195302 (+ 1670us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:52.195715 (+ 413us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.195754 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:52.196305 (+ 551us) log.cc:842] Serialized 26 byte log entry
0806 04:07:53.194318 (+998013us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:53.194380 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.194469 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:53.575730 (+381261us) raft_consensus.cc:1671] finished
0806 04:07:53.575749 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.575889 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98514,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":174,"thread_start_us":985576,"threads_started":1,"wal-append.queue_time_us":1369247}
W20260806 04:07:53.594894 5037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.009s sys 0.032s
W20260806 04:07:53.824973 5037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.174s user 0.009s sys 0.032s
I20260806 04:07:53.580647 5038 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 5. Ops: []
I20260806 04:07:53.827980 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 191) took 4846 ms. Trace:
I20260806 04:07:53.828425 5038 rpcz_store.cc:276] 0806 04:07:48.981721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.485997 (+4504276us) service_pool.cc:227] Handling call
0806 04:07:53.580475 (+ 94478us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.827211 (+246736us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.827672 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:53.732764 4996 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.908s user 0.003s sys 0.042s
W20260806 04:07:53.829834 4996 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.006s user 0.003s sys 0.042s
W20260806 04:07:53.830451 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.00648e+06us
I20260806 04:07:53.831935 4816 raft_consensus.cc:2396] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:07:53.833380 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:53.567981 5028 rpcz_store.cc:276] 0806 04:07:50.059506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.483645 (+3424139us) service_pool.cc:227] Handling call
0806 04:07:53.484012 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.487514 (+ 3502us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.487556 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:53.487612 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:53.487660 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.487797 (+ 137us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.487951 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2960,"raft.queue_time_us":8830,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":282}
I20260806 04:07:53.835212 7603 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:53.566953 5030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.028s sys 0.037s
W20260806 04:07:53.836207 5030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.203s user 0.029s sys 0.037s
I20260806 04:07:53.836244 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:53.836127 7603 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:53.608561 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1895) took 4006 ms. Trace:
I20260806 04:07:53.839040 5026 rpcz_store.cc:276] 0806 04:07:49.601844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.480486 (+3878642us) service_pool.cc:227] Handling call
0806 04:07:53.575984 (+ 95498us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.608181 (+ 32197us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.608475 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:53.839676 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 108471us
I20260806 04:07:53.557929 5029 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:53.581223 5036 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:07:53.753139 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:53.754560 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:53.754887 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:53.755379 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:53.756965 7852 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:53.757359 5866 raft_consensus.cc:1226] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:53.759080 7586 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:53.640604 7762 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:53.643445 7830 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:53.659644 8002 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:53.659981 8008 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:53.796571 4809 rpcz_store.cc:269] 0806 04:07:51.300675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.404157 (+103482us) service_pool.cc:227] Handling call
0806 04:07:53.700274 (+2296117us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003fcd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.700762 (+ 488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3051268992}
I20260806 04:07:53.799796 4813 rpcz_store.cc:276] 0806 04:07:52.076308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.469788 (+393480us) service_pool.cc:227] Handling call
0806 04:07:52.470220 (+ 432us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.898129 (+427909us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.720066 (+821937us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5400231c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:53.720533 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":945030528}
W20260806 04:07:53.808549 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.886s user 0.017s sys 0.020s
I20260806 04:07:53.809811 4920 rpcz_store.cc:276] 0806 04:07:51.942352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.942522 (+ 170us) service_pool.cc:227] Handling call
0806 04:07:53.491641 (+1549119us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:53.575745 (+ 84104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13218560}
W20260806 04:07:53.797443 4810 rpcz_store.cc:269] 0806 04:07:51.228294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:51.569198 (+340904us) service_pool.cc:227] Handling call
0806 04:07:53.707633 (+2138435us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54003b06d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.707915 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1015127424}
W20260806 04:07:53.816473 4924 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.183s user 0.007s sys 0.028s
W20260806 04:07:53.811447 4923 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s user 0.010s sys 0.035s
W20260806 04:07:53.823094 4931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.865s user 0.003s sys 0.025s
W20260806 04:07:53.819944 4878 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.857s user 0.002s sys 0.019s
I20260806 04:07:53.838356 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:53.840303 7816 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:07:53.839699 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:53.843381 5034 raft_consensus.cc:1261] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:53.840351 4669 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46902 (request call id 283) took 9919 ms. Trace:
I20260806 04:07:53.814175 7714 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:53.797116 4817 rpcz_store.cc:276] 0806 04:07:51.813234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.315715 (+502481us) service_pool.cc:227] Handling call
0806 04:07:52.316056 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:52.316272 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.316302 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:52.321219 (+ 4917us) log.cc:842] Serialized 26 byte log entry
0806 04:07:53.339564 (+1018345us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:53.339616 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.339706 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:53.706924 (+367218us) raft_consensus.cc:1671] finished
0806 04:07:53.706995 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.707433 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1010835,"threads_started":1,"wal-append.queue_time_us":1307753}
I20260806 04:07:53.840606 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1282) took 1036 ms. Trace:
I20260806 04:07:53.850311 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:53.855664 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2019) took 2040 ms. Trace:
I20260806 04:07:53.915827 4804 rpcz_store.cc:276] 0806 04:07:51.815469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.330876 (+515407us) service_pool.cc:227] Handling call
0806 04:07:53.708693 (+1377817us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.803146 (+ 94453us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.851909 (+ 48763us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54003b06d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:53.855581 (+ 3672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3167486336}
I20260806 04:07:53.917371 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 316) took 1286 ms. Trace:
W20260806 04:07:53.918797 7419 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.009s sys 0.017s
W20260806 04:07:53.919337 7419 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.009s sys 0.017s
I20260806 04:07:53.919785 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1587) took 1014 ms. Trace:
I20260806 04:07:53.920303 4931 rpcz_store.cc:276] 0806 04:07:52.905570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.956214 (+ 50644us) service_pool.cc:227] Handling call
0806 04:07:52.956707 (+ 493us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.956944 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:07:52.957949 (+ 1005us) log.cc:842] Serialized 14 byte log entry
0806 04:07:53.919076 (+961127us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:53.919204 (+ 128us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:53.919355 (+ 151us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.919502 (+ 147us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.919702 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":366,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":73,"raft.queue_time_us":1251,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":136,"thread_start_us":954121,"threads_started":1,"wal-append.queue_time_us":924643}
W20260806 04:07:53.923807 4876 leader_election.cc:334] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.630s (SENT)
I20260806 04:07:53.924568 4876 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:53.927170 4925 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:53.929033 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:53.930605 8008 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:53.930965 7816 raft_consensus.cc:686] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:53.931391 8008 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:53.932780 7816 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:53.904874 4919 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.850549 7603 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:53.839696 4668 catalog_manager.cc:5495] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:53.872380 5856 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.873060 5859 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:53.874701 5026 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 04:07:53.873229 7586 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:53.875522 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 206) took 1066 ms. Trace:
I20260806 04:07:53.875306 7852 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:53.883076 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 49398.7us
I20260806 04:07:53.862383 4805 rpcz_store.cc:276] 0806 04:07:52.198955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.410688 (+1211733us) service_pool.cc:227] Handling call
0806 04:07:53.411084 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.411322 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.411369 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:53.411421 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:53.411467 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.411544 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.411720 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.888053 5038 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:07:53.891088 4928 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:07:53.915161 4814 rpcz_store.cc:276] 0806 04:07:52.785697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.387535 (+601838us) service_pool.cc:227] Handling call
0806 04:07:53.818770 (+431235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.917923 4809 rpcz_store.cc:276] 0806 04:07:52.631372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.915905 (+1284533us) service_pool.cc:227] Handling call
0806 04:07:53.916456 (+ 551us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.916910 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.916937 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:53.917001 (+ 64us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:53.917052 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.917126 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:53.917277 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.940415 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:53.945765 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1617) took 2671 ms. Trace:
W20260806 04:07:53.942736 7609 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.018s sys 0.045s
W20260806 04:07:53.947459 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.02148e+06us
I20260806 04:07:53.953876 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 194) took 1162 ms. Trace:
I20260806 04:07:53.978955 5026 rpcz_store.cc:276] 0806 04:07:52.791729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.840184 (+1048455us) service_pool.cc:227] Handling call
0806 04:07:53.847212 (+ 7028us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:53.949151 (+101939us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.953794 (+ 4643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.953339 5029 rpcz_store.cc:276] 0806 04:07:52.794548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.426100 (+631552us) service_pool.cc:227] Handling call
0806 04:07:53.858667 (+432567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.977438 5034 rpcz_store.cc:276] 0806 04:07:51.270936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.825155 (+2554219us) service_pool.cc:227] Handling call
0806 04:07:53.825976 (+ 821us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.914383 (+ 88407us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.941890 (+ 27507us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:53.947530 7875 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.008s sys 0.031s
I20260806 04:07:53.940197 8008 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:53.946844 7932 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.050s sys 0.087s
I20260806 04:07:53.914753 4669 rpcz_store.cc:276] 0806 04:07:43.921142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:43.926778 (+ 5636us) service_pool.cc:227] Handling call
0806 04:07:53.840054 (+9913276us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:47.316111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:47.672854 (+356743us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:47.690255 (+ 17401us) write_op.cc:435] Acquired schema lock
0806 04:07:47.700709 (+ 10454us) tablet.cc:660] Decoding operations
0806 04:07:47.746536 (+ 45827us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:47.746711 (+ 175us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:47.746743 (+ 32us) tablet.cc:683] Acquiring locks for 3 operations
0806 04:07:47.794768 (+ 48025us) tablet.cc:699] Row locks acquired
0806 04:07:47.802801 (+ 8033us) write_op.cc:260] PREPARE: finished
0806 04:07:47.830983 (+ 28182us) write_op.cc:270] Start()
0806 04:07:47.831211 (+ 228us) write_op.cc:276] Timestamp: P: 1785989267830891 usec, L: 0
0806 04:07:47.831264 (+ 53us) op_driver.cc:331] REPLICATION: starting
0806 04:07:47.885874 (+ 54610us) log.cc:842] Serialized 1303 byte log entry
0806 04:07:49.376999 (+1491125us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:50.015060 (+638061us) op_driver.cc:444] REPLICATION: finished
0806 04:07:50.957644 (+942584us) write_op.cc:301] APPLY: starting
0806 04:07:50.959799 (+ 2155us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:50.979079 (+ 19280us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:50.982014 (+ 2935us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:51.122205 (+140191us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:51.163558 (+ 41353us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 04:07:51.179103 (+ 15545us) write_op.cc:312] APPLY: finished
0806 04:07:51.196825 (+ 17722us) log.cc:842] Serialized 35 byte log entry
0806 04:07:51.553276 (+356451us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:51.576641 (+ 23365us) write_op.cc:454] Released schema lock
0806 04:07:51.591476 (+ 14835us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:52.306502 (+715026us) spinlock_profiling.cc:243] Waited 5.66 ms on lock 0x7b54001c1528. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23dfe7be91 00007f23dfe7aafc 00007f23dd37f0dd 00007f23dd36b832 00007f23dd36afd1 00007f23dd36edd1 00007f23dd36ed89 00007f23dd36ed19 00007f23dd36ece1 00007f23dd36dfdd 00007f23dfc96204
0806 04:07:52.307009 (+ 507us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:52.307063 (+ 54us) write_op.cc:454] Released schema lock
0806 04:07:53.255359 (+948296us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":897359,"apply.run_cpu_time_us":20535,"apply.run_wall_time_us":1370129,"num_ops":3,"prepare.queue_time_us":380463,"prepare.run_cpu_time_us":27171,"prepare.run_wall_time_us":1048587,"raft.queue_time_us":347675,"raft.run_cpu_time_us":234502,"raft.run_wall_time_us":561825,"replication_time_us":2162241,"spinlock_wait_cycles":884859776,"thread_start_us":1599492,"threads_started":5,"wal-append.queue_time_us":1163040,"wal-append.run_cpu_time_us":86240,"wal-append.run_wall_time_us":2315343}]]}
I20260806 04:07:53.989804 4668 catalog_manager.cc:5495] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:53.984614 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:53.993309 5034 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:53.996064 5029 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:53.998762 4813 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:53.985323 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1618) took 2706 ms. Trace:
I20260806 04:07:54.010398 5038 rpcz_store.cc:276] 0806 04:07:51.271245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.829451 (+2558206us) service_pool.cc:227] Handling call
0806 04:07:53.847053 (+ 17602us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.975656 (+128603us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.978016 (+ 2360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:53.949589 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:53.981871 4876 leader_election.cc:334] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.927s (SENT)
I20260806 04:07:53.991477 7586 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:54.004089 7875 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s user 0.009s sys 0.031s
I20260806 04:07:54.012372 7643 consensus_queue.cc:1046] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 04:07:54.012984 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 318) took 1109 ms. Trace:
W20260806 04:07:53.999055 5812 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.018s sys 0.021s
I20260806 04:07:54.013352 4925 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:54.013823 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 200) took 1213 ms. Trace:
I20260806 04:07:54.016532 4810 rpcz_store.cc:276] 0806 04:07:52.903942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.934598 (+1030656us) service_pool.cc:227] Handling call
0806 04:07:53.999594 (+ 64996us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.009378 (+ 9784us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:54.009414 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:54.012688 (+ 3274us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:54.012736 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.012799 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.012911 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.019814 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 201) took 1220 ms. Trace:
I20260806 04:07:54.020394 5029 rpcz_store.cc:276] 0806 04:07:52.793395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.995191 (+1201796us) service_pool.cc:227] Handling call
0806 04:07:53.995720 (+ 529us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.009829 (+ 14109us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.013597 (+ 3768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.019869 5034 rpcz_store.cc:276] 0806 04:07:52.793161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.992351 (+1199190us) service_pool.cc:227] Handling call
0806 04:07:53.992895 (+ 544us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.005071 (+ 12176us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.006900 (+ 1829us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:54.024730 5812 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.059s user 0.018s sys 0.022s
W20260806 04:07:54.022629 5863 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.001s sys 0.019s
W20260806 04:07:54.009408 7932 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s user 0.050s sys 0.088s
I20260806 04:07:54.034322 4876 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:54.016944 7888 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.006s sys 0.021s
W20260806 04:07:54.037530 4768 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.089s user 0.010s sys 0.009s
I20260806 04:07:54.046773 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:54.040881 5035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.020s sys 0.036s
I20260806 04:07:54.037896 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1908) took 3357 ms. Trace:
W20260806 04:07:54.054436 4768 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.239s user 0.010s sys 0.009s
I20260806 04:07:54.056092 4927 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
W20260806 04:07:54.046113 4818 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.004s sys 0.008s
I20260806 04:07:54.058167 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:54.053352 5035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.020s sys 0.036s
I20260806 04:07:54.047613 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:54.054963 5863 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s user 0.002s sys 0.019s
I20260806 04:07:54.054193 5036 rpcz_store.cc:276] 0806 04:07:50.546993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.580518 (+3033525us) service_pool.cc:227] Handling call
0806 04:07:53.580851 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.873543 (+292692us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.904188 (+ 30645us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:54.072228 4818 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s user 0.005s sys 0.009s
I20260806 04:07:54.078116 7720 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:54.079962 4926 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:54.062840 7888 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s user 0.006s sys 0.022s
I20260806 04:07:54.078025 5866 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
W20260806 04:07:54.087736 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 265222us
I20260806 04:07:54.070179 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.069181 7936 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.091228 4925 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:54.085526 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.18238e+06us
I20260806 04:07:54.100085 4817 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:54.107290 4931 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:54.115724 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 319) took 1193 ms. Trace:
I20260806 04:07:54.116254 4817 rpcz_store.cc:276] 0806 04:07:52.917126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.976937 (+1059811us) service_pool.cc:227] Handling call
0806 04:07:54.053971 (+ 77034us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.108634 (+ 54663us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.110643 (+ 2009us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.121940 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2031) took 1334 ms. Trace:
I20260806 04:07:54.122366 4813 rpcz_store.cc:276] 0806 04:07:52.787842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.920540 (+1132698us) service_pool.cc:227] Handling call
0806 04:07:53.920925 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.013745 (+ 92820us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.120165 (+106420us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b54002710d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:54.121872 (+ 1707us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50602368}
I20260806 04:07:54.121717 7419 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:54.128326 7419 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.132632 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1576) took 1323 ms. Trace:
W20260806 04:07:54.122979 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.30798e+06us
I20260806 04:07:54.133201 5038 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 04:07:54.144852 5034 raft_consensus.cc:1226] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:54.148819 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1622) took 1329 ms. Trace:
I20260806 04:07:54.142727 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 205) took 1348 ms. Trace:
I20260806 04:07:54.149250 5034 rpcz_store.cc:276] 0806 04:07:52.819454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.040387 (+1220933us) service_pool.cc:227] Handling call
0806 04:07:54.097106 (+ 56719us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.148514 (+ 51408us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.148753 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.149468 5038 rpcz_store.cc:276] 0806 04:07:52.794319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.036857 (+1242538us) service_pool.cc:227] Handling call
0806 04:07:54.100801 (+ 63944us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:54.142332 (+ 41531us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.142631 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.154261 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33546 (request call id 695) took 1503 ms. Trace:
I20260806 04:07:54.160866 7852 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.163466 5859 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:54.166774 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:54.167223 7720 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:54.167739 7720 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:54.171077 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 226316us
I20260806 04:07:54.156214 5029 raft_consensus.cc:1226] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 04:07:54.158156 4809 rpcz_store.cc:276] 0806 04:07:52.629416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.980902 (+1351486us) service_pool.cc:227] Handling call
0806 04:07:54.132300 (+151398us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:07:54.152992 8020 log.cc:924] Time spent T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.139s user 0.007s sys 0.003s
I20260806 04:07:54.174026 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1621) took 1355 ms. Trace:
I20260806 04:07:54.177939 5029 rpcz_store.cc:276] 0806 04:07:52.819085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.040448 (+1221363us) service_pool.cc:227] Handling call
0806 04:07:54.107604 (+ 67156us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:54.172385 (+ 64781us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.173962 (+ 1577us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.156072 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:54.183341 8034 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260806 04:07:54.184198 5856 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:07:54.186604 4876 leader_election.cc:334] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.975s (SENT)
I20260806 04:07:54.187222 4876 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:54.189833 5034 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:07:54.190896 7932 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:54.193794 7594 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:54.193953 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.194612 7830 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:54.195273 7419 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:54.195351 8010 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.196214 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:54.197131 4809 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:54.199616 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1633) took 1376 ms. Trace:
I20260806 04:07:54.200284 4879 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.201036 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 314) took 1815 ms. Trace:
I20260806 04:07:54.141472 4923 rpcz_store.cc:276] 0806 04:07:52.799767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.799922 (+ 155us) service_pool.cc:227] Handling call
0806 04:07:52.867887 (+ 67965us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.868172 (+ 285us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:07:52.868551 (+ 379us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.052277 (+1183726us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:54.072224 (+ 19947us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:54.086054 (+ 13830us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.095278 (+ 9224us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.122780 (+ 27502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104373,"prepare.queue_time_us":3426,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":79,"raft.queue_time_us":107302,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":190,"thread_start_us":1043371,"threads_started":1,"wal-append.queue_time_us":714461}
I20260806 04:07:54.201443 4808 rpcz_store.cc:276] 0806 04:07:52.385021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.419536 (+1034515us) service_pool.cc:227] Handling call
0806 04:07:53.419987 (+ 451us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:07:53.698098 (+278111us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.698224 (+ 126us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:07:53.700034 (+ 1810us) log.cc:842] Serialized 52 byte log entry
0806 04:07:53.972191 (+272157us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:53.986083 (+ 13892us) log.cc:842] Serialized 14 byte log entry
0806 04:07:53.986412 (+ 329us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.986579 (+ 167us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.200587 (+214008us) raft_consensus.cc:1671] finished
0806 04:07:54.200607 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.200814 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13698,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":2465,"raft.run_cpu_time_us":540,"raft.run_wall_time_us":12290,"thread_start_us":249055,"threads_started":1,"wal-append.queue_time_us":453931}
I20260806 04:07:54.203478 7932 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:54.204404 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:54.205407 4813 raft_consensus.cc:2396] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:07:54.207050 5866 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:54.207859 7586 raft_consensus.cc:3032] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:07:54.208395 7586 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:54.208706 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:54.211825 5862 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 6. Ops: [4.6-4.6]
I20260806 04:07:54.212389 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:54.211766 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:54.216066 5859 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:54.215862 4668 catalog_manager.cc:5495] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:54.220242 5867 raft_consensus.cc:1226] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:54.209677 7586 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.207970 5856 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:54.200107 5029 rpcz_store.cc:276] 0806 04:07:52.822832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.198353 (+1375521us) service_pool.cc:227] Handling call
0806 04:07:54.198838 (+ 485us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.199147 (+ 309us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:54.199184 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:54.199331 (+ 147us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:54.199372 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.199451 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.199557 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.171079 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2014) took 2357 ms. Trace:
I20260806 04:07:54.217895 4668 catalog_manager.cc:5495] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 3 to 4, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 4 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
W20260806 04:07:54.200331 7896 consensus_meta.cc:287] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.558s user 0.006s sys 0.007s
I20260806 04:07:54.190582 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1625) took 1370 ms. Trace:
I20260806 04:07:54.087658 7762 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:54.223151 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 71188.4us
I20260806 04:07:54.234134 5034 rpcz_store.cc:276] 0806 04:07:52.820366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.159580 (+1339214us) service_pool.cc:227] Handling call
0806 04:07:54.189247 (+ 29667us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.190269 (+ 1022us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.190507 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.233202 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 186) took 5245 ms. Trace:
I20260806 04:07:54.237880 5037 rpcz_store.cc:276] 0806 04:07:48.980195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.558889 (+3578694us) service_pool.cc:227] Handling call
0806 04:07:52.559359 (+ 470us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:52.593225 (+ 33866us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.593272 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:52.593916 (+ 644us) log.cc:842] Serialized 26 byte log entry
0806 04:07:53.825943 (+1232027us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:54.087510 (+261567us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.118599 (+ 31089us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.124216 (+ 5617us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.124299 (+ 83us) raft_consensus.cc:1671] finished
0806 04:07:54.124317 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.225945 (+101628us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7748,"prepare.queue_time_us":403583,"raft.queue_time_us":31998,"raft.run_cpu_time_us":410,"raft.run_wall_time_us":38924,"thread_start_us":1433093,"threads_started":2,"wal-append.queue_time_us":951043}
I20260806 04:07:54.236649 7643 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:54.230674 4818 rpcz_store.cc:276] 0806 04:07:51.813861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.320699 (+506838us) service_pool.cc:227] Handling call
0806 04:07:52.321054 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:52.321305 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:07:53.265775 (+944470us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.132559 (+866784us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:54.153105 (+ 20546us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:54.159060 (+ 5955us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.170835 (+ 11775us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.170996 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":366194,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":13451,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":40763,"thread_start_us":1729422,"threads_started":2,"wal-append.queue_time_us":771389}
I20260806 04:07:54.250943 5860 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:54.254897 7889 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:07:54.255046 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 53448.4us
I20260806 04:07:54.257457 5038 raft_consensus.cc:1226] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
W20260806 04:07:54.257534 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 55450.2us
I20260806 04:07:54.259274 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:54.258277 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:54.253861 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2025) took 1551 ms. Trace:
I20260806 04:07:54.263887 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:54.265426 4931 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:54.248117 7419 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:54.248592 7896 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.269629 7714 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.269819 7896 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.270301 4770 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.270795 4805 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
W20260806 04:07:54.271205 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 49116.7us
I20260806 04:07:54.272065 4770 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.273902 7587 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:54.274571 7587 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:54.275918 7587 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.278817 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:54.279785 5857 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:54.279268 7896 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:54.273545 4668 catalog_manager.cc:5495] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:07:54.282939 7594 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:54.283488 7594 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:54.287911 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:54.288930 4926 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:54.289827 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:54.270047 5859 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:07:54.261781 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:54.290557 4926 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:54.301689 4928 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:54.269846 4882 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.261041 4804 rpcz_store.cc:276] 0806 04:07:52.699042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.917018 (+1217976us) service_pool.cc:227] Handling call
0806 04:07:53.917441 (+ 423us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:53.917918 (+ 477us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:07:53.918520 (+ 602us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.131444 (+212924us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b180030a300. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32638 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:07:54.248677 (+117233us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:54.248910 (+ 233us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:54.249062 (+ 152us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.249454 (+ 392us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.249814 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7789,"prepare.queue_time_us":29027,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":15019,"spinlock_wait_cycles":9390848,"thread_start_us":281899,"threads_started":1}
W20260806 04:07:54.307116 8043 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260806 04:07:54.330602 7994 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:54.330433 7643 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.331447 7876 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:54.331985 7876 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:07:54.321734 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 58180.4us
I20260806 04:07:54.332257 7994 raft_consensus.cc:686] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.332352 7876 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:54.333039 7643 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.333259 4882 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.333463 7994 consensus_queue.cc:237] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.335043 7936 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:54.335610 7936 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.336676 7936 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.330729 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:54.331537 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1560) took 4363 ms. Trace:
I20260806 04:07:54.322582 7419 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.343410 7419 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:54.342309 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 89450us
W20260806 04:07:54.344452 7754 consensus_meta.cc:287] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.695s user 0.011s sys 0.006s
I20260806 04:07:54.345285 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 189) took 5364 ms. Trace:
I20260806 04:07:54.345018 7754 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.348757 5854 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:54.349539 7908 consensus_queue.cc:1046] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:54.351332 7908 raft_consensus.cc:3032] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:54.351821 7908 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.353145 7908 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.355248 5034 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:07:54.355908 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 209) took 1448 ms. Trace:
I20260806 04:07:54.356215 5034 rpcz_store.cc:276] 0806 04:07:52.907487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.249425 (+1341938us) service_pool.cc:227] Handling call
0806 04:07:54.306977 (+ 57552us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.355645 (+ 48668us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.355853 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:54.357125 7721 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:54.357800 7721 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.360028 7721 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:54.360879 7754 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.266805 7830 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260806 04:07:54.343705 7720 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.369426 7720 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.339196 4924 rpcz_store.cc:276] 0806 04:07:49.967791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:49.979040 (+ 11249us) service_pool.cc:227] Handling call
0806 04:07:52.483915 (+2504875us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:52.484185 (+ 270us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:52.484605 (+ 420us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.484640 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:52.589133 (+104493us) log.cc:842] Serialized 26 byte log entry
0806 04:07:53.935391 (+1346258us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:53.935484 (+ 93us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:53.935599 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.331191 (+395592us) raft_consensus.cc:1671] finished
0806 04:07:54.331211 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.331450 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11629,"raft.queue_time_us":214019,"raft.run_cpu_time_us":9779,"raft.run_wall_time_us":61285,"spinlock_wait_cycles":4278144,"thread_start_us":1301801,"threads_started":1,"wal-append.queue_time_us":1284107}
W20260806 04:07:54.371428 8036 log.cc:924] Time spent T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20260806 04:07:54.366189 5032 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:07:54.373536 8002 consensus_queue.cc:1046] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:54.376230 8002 raft_consensus.cc:3032] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:54.376652 8002 raft_consensus.cc:729] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.377828 8002 consensus_queue.cc:260] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.380061 4925 raft_consensus.cc:2396] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 04:07:54.367208 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 56914.1us
I20260806 04:07:54.382566 4923 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:54.311770 7586 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:54.383528 7762 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.308055 7889 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:54.446241 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 72736.1us
I20260806 04:07:54.423907 7896 raft_consensus.cc:686] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:54.443848 4992 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:54.424068 4919 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:54.444262 7908 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.445154 7754 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:54.443979 7721 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:54.477139 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.423048 7594 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.433826 7587 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
W20260806 04:07:54.480189 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 58468.8us
W20260806 04:07:54.483939 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 468291us
I20260806 04:07:54.495914 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1632) took 1673 ms. Trace:
I20260806 04:07:54.496371 5038 rpcz_store.cc:276] 0806 04:07:52.822487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.155877 (+1333390us) service_pool.cc:227] Handling call
0806 04:07:54.204623 (+ 48746us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.261280 (+ 56657us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.495561 (+234281us) spinlock_profiling.cc:243] Waited 52.5 ms on lock 0x7b540037df54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:54.495838 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120856832}
I20260806 04:07:54.501593 7609 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 04:07:54.502154 8065 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 04:07:54.502537 7609 consensus_queue.cc:1225] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260806 04:07:54.502697 7877 consensus_meta.cc:287] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.847s user 0.013s sys 0.005s
I20260806 04:07:54.503751 7720 raft_consensus.cc:3032] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:54.504257 7720 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:07:54.504488 8057 log.cc:924] Time spent T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260806 04:07:54.505462 5863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 534) took 1243 ms. Trace:
I20260806 04:07:54.505369 7720 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.505913 5863 rpcz_store.cc:276] 0806 04:07:53.262365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.262555 (+ 190us) service_pool.cc:227] Handling call
0806 04:07:53.263024 (+ 469us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:53.263282 (+ 258us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.263317 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:53.263851 (+ 534us) log.cc:842] Serialized 26 byte log entry
0806 04:07:54.143088 (+879237us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:54.153207 (+ 10119us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.173520 (+ 20313us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.505170 (+331650us) raft_consensus.cc:1671] finished
0806 04:07:54.505189 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.505363 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":723893,"threads_started":1,"wal-append.queue_time_us":1199552}
I20260806 04:07:54.509486 4919 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:54.509660 7889 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.470398 7419 consensus_queue.cc:1046] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 04:07:54.511190 7419 raft_consensus.cc:3032] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:54.511632 7419 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.483942 7752 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.512643 7419 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.514139 7419 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.514899 4992 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:54.516230 4992 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:54.516453 8068 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:54.516999 8068 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.517122 7603 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:54.517470 8068 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.475090 7754 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.519265 4996 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.520939 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.521415 7754 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.492218 7721 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.524902 7721 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.525172 7816 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.518718 7762 raft_consensus.cc:2781] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:54.525787 7762 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.526197 7762 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.345762 5030 rpcz_store.cc:276] 0806 04:07:48.981148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:52.596104 (+3614956us) service_pool.cc:227] Handling call
0806 04:07:52.596534 (+ 430us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:52.597014 (+ 480us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:52.597054 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:52.597606 (+ 552us) log.cc:842] Serialized 26 byte log entry
0806 04:07:53.837121 (+1239515us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:54.084685 (+247564us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.313086 (+228401us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.344863 (+ 31777us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.344968 (+ 105us) raft_consensus.cc:1671] finished
0806 04:07:54.344984 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.345205 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57128,"prepare.queue_time_us":422943,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":46072,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":38863,"thread_start_us":1627494,"threads_started":3,"wal-append.queue_time_us":947738}
I20260806 04:07:54.522936 7830 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:54.530342 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1869) took 5607 ms (client timeout 2018 ms). Trace:
W20260806 04:07:54.533360 5032 rpcz_store.cc:269] 0806 04:07:48.923364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.110947 (+1187583us) service_pool.cc:227] Handling call
0806 04:07:54.530054 (+4419107us) spinlock_profiling.cc:243] Waited 4.15 s on lock 0x7b54003fc0d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:54.530249 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9548023680}
I20260806 04:07:54.548231 4992 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:54.544288 7594 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.554405 4996 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.554688 5028 raft_consensus.cc:1261] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 04:07:54.565237 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 248166us
I20260806 04:07:54.566989 7594 consensus_queue.cc:1046] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:54.568709 7979 raft_consensus.cc:3032] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:54.569197 7979 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.571020 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:54.572714 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:54.573006 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:54.573632 4814 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:07:54.573830 4996 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:54.574542 7830 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:07:54.575122 7603 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:54.575574 7603 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.570461 7979 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.576113 7603 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:54.576241 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 45147.3us
I20260806 04:07:54.576802 7979 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:54.577705 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:54.574847 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 43609.8us
I20260806 04:07:54.580739 4992 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:54.581791 8069 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:54.581557 8092 consensus_queue.cc:1046] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:54.582492 8069 raft_consensus.cc:686] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:54.582784 8092 raft_consensus.cc:3032] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:54.583243 8092 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:54.583634 8069 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.584193 8092 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.584815 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:54.585515 4923 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:54.585666 8092 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.585737 7603 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.586730 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:54.587008 8087 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 04:07:54.587809 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:54.588157 7603 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.588518 4924 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.575418 4996 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.573922 4805 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:54.571189 8067 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:54.591249 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1914) took 1894 ms. Trace:
I20260806 04:07:54.591958 4992 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:54.554425 8002 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:54.586530 7875 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.010s
I20260806 04:07:54.606443 5816 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.608045 4879 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:54.596061 8091 log.cc:924] Time spent T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260806 04:07:54.612465 7603 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:54.615414 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 196) took 1823 ms. Trace:
I20260806 04:07:54.615916 5026 rpcz_store.cc:276] 0806 04:07:52.792228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.009560 (+1217332us) service_pool.cc:227] Handling call
0806 04:07:54.013308 (+ 3748us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:54.106892 (+ 93584us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:54.106929 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:54.150956 (+ 44027us) log.cc:842] Serialized 26 byte log entry
0806 04:07:54.327743 (+176787us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:54.327782 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.327891 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.615130 (+287239us) raft_consensus.cc:1671] finished
0806 04:07:54.615156 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.615343 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6020,"raft.queue_time_us":19308,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":7687,"thread_start_us":145563,"threads_started":1,"wal-append.queue_time_us":430979}
I20260806 04:07:54.597837 5028 rpcz_store.cc:276] 0806 04:07:52.662290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.835286 (+1172996us) service_pool.cc:227] Handling call
0806 04:07:54.530791 (+695505us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:54.555156 (+ 24365us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.556783 (+ 1627us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54003fc0d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:54.556922 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1598491904}
I20260806 04:07:54.626920 7816 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:54.627507 7816 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.627900 7816 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.594766 7889 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
W20260806 04:07:54.568646 5812 connection.cc:232] RPC call timeout handler was delayed by -1.3304847257677466s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20260806 04:07:54.631000 7889 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:54.631333 5812 connection.cc:232] RPC call timeout handler was delayed by -1.3304847257677466s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
W20260806 04:07:54.631891 5812 connection.cc:232] RPC call timeout handler was delayed by -1.2944847259204835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22320000000000004s for a response
W20260806 04:07:54.632653 5812 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.570s (SENT) [suppressed 37 similar messages]
I20260806 04:07:54.632215 7889 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.634126 7889 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:07:54.643610 8073 log.cc:924] Time spent T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260806 04:07:54.530400 7896 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.649168 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 193) took 5104 ms. Trace:
I20260806 04:07:54.649788 5035 rpcz_store.cc:276] 0806 04:07:49.544605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.476964 (+3932359us) service_pool.cc:227] Handling call
0806 04:07:53.477302 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:53.477985 (+ 683us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.478010 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:53.478494 (+ 484us) log.cc:842] Serialized 26 byte log entry
0806 04:07:54.148839 (+670345us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:54.433353 (+284514us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.452303 (+ 18950us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.460953 (+ 8650us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.648882 (+187929us) raft_consensus.cc:1671] finished
0806 04:07:54.648913 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.649087 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35502,"prepare.queue_time_us":96534,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":8681,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":10315,"thread_start_us":765988,"threads_started":2,"wal-append.queue_time_us":1167213}
I20260806 04:07:54.623034 7876 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.654838 8078 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:54.655480 8078 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.655917 8078 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.656396 7876 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.658001 7754 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:54.658769 7754 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.575512 7830 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:54.661859 7754 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:54.662369 7830 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:54.662968 7830 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:54.675215 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 65283.7us
I20260806 04:07:54.672622 4924 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
W20260806 04:07:54.678987 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 72592.3us
W20260806 04:07:54.685571 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 110260us
W20260806 04:07:54.683998 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 196668us
I20260806 04:07:54.687271 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:54.610337 8067 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.692319 5816 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.688666 7936 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.689123 7875 raft_consensus.cc:3032] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:07:54.682466 7721 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:07:54.577062 5857 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
W20260806 04:07:54.688570 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 91344.4us
I20260806 04:07:54.588348 8087 consensus_queue.cc:1046] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.095s
I20260806 04:07:54.700663 8085 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:54.701251 8085 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.701671 8085 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:54.706156 5812 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.710628 7603 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:07:54.711153 7603 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:54.711591 7603 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:07:54.712038 7720 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.715116 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.639221 7762 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.696147 7875 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:54.698319 7721 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.734885 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:54.738128 4804 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:54.745272 7888 raft_consensus.cc:3032] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:54.745903 7888 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:54.753103 4928 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:54.757915 4809 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:54.759339 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:54.760325 4818 raft_consensus.cc:2396] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:07:54.761238 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:54.762573 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:54.764856 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1623) took 1945 ms. Trace:
I20260806 04:07:54.765251 5036 rpcz_store.cc:276] 0806 04:07:52.819761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.096973 (+1277212us) service_pool.cc:227] Handling call
0806 04:07:54.144360 (+ 47387us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:54.202995 (+ 58635us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:54.203027 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:54.203640 (+ 613us) log.cc:842] Serialized 26 byte log entry
0806 04:07:54.552488 (+348848us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:07:54.552545 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.552642 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.764640 (+211998us) raft_consensus.cc:1671] finished
0806 04:07:54.764659 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.764801 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45436,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":10938,"thread_start_us":348308,"threads_started":1,"wal-append.queue_time_us":486716}
W20260806 04:07:54.768095 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 80924us
W20260806 04:07:54.770557 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 48300.8us
I20260806 04:07:54.778534 4814 raft_consensus.cc:1261] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 04:07:54.780947 7714 consensus_meta.cc:287] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.511s user 0.005s sys 0.008s
I20260806 04:07:54.723616 7936 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:54.813076 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 315) took 2337 ms. Trace:
I20260806 04:07:54.811733 4930 raft_consensus.cc:2396] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:07:54.740772 7721 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:07:54.813715 4816 rpcz_store.cc:276] 0806 04:07:52.475313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.856821 (+1381508us) service_pool.cc:227] Handling call
0806 04:07:53.872788 (+ 15967us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:53.874322 (+ 1534us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:53.874360 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:53.874956 (+ 596us) log.cc:842] Serialized 26 byte log entry
0806 04:07:54.089574 (+214618us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:07:54.089637 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.089722 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:54.812771 (+723049us) raft_consensus.cc:1671] finished
0806 04:07:54.812803 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:54.812986 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":214156,"threads_started":1,"wal-append.queue_time_us":814018}
I20260806 04:07:54.782433 4768 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:54.778586 8094 consensus_queue.cc:1046] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
W20260806 04:07:54.828488 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 130914us
I20260806 04:07:54.810799 7936 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.781668 7816 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.760344 7830 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:54.865531 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 129683us
I20260806 04:07:54.873066 4996 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.873709 4804 raft_consensus.cc:1261] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:54.856010 7936 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:54.854712 7852 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:54.891153 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 108818us
I20260806 04:07:54.897107 5812 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:54.882746 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.907618 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:54.908806 7603 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.911172 7762 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:54.911470 8090 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:54.911865 7762 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.912164 8087 consensus_queue.cc:1035] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:54.877737 7876 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.921360 7586 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:54.921569 8094 consensus_queue.cc:1225] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:07:54.922143 8094 raft_consensus.cc:3032] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:07:54.922031 7586 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.922511 8094 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:54.923363 8094 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.924335 7586 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:54.924583 8094 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:54.925112 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:54.925715 5863 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.911885 8090 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:54.887231 8084 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:07:54.927080 8090 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:54.891333 8067 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:54.938504 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 172179us
W20260806 04:07:54.940479 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 67523us
I20260806 04:07:54.943059 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1449) took 1098 ms. Trace:
I20260806 04:07:54.944152 4930 rpcz_store.cc:276] 0806 04:07:53.836402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:53.839183 (+ 2781us) service_pool.cc:227] Handling call
0806 04:07:54.935086 (+1095903us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b5400288054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:54.935226 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1999222144}
I20260806 04:07:54.864826 7875 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.948426 7603 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.929862 7852 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.960785 8067 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:54.975507 4996 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:54.978472 7714 consensus_queue.cc:1035] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:54.992537 7852 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.993014 7714 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:54.993834 7721 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.995414 8068 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:54.996340 7721 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:07:54.993772 7714 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:54.997890 8068 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:54.998311 7714 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:54.881846 7888 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:55.000363 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 133952us
I20260806 04:07:54.980202 7594 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:07:54.981389 8098 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260806 04:07:54.984127 4816 raft_consensus.cc:1261] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 04:07:54.986434 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 77443.3us
I20260806 04:07:54.999840 7888 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:55.012181 5812 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.015355 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 312) took 1090 ms. Trace:
I20260806 04:07:54.968552 7877 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:54.975463 8002 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.035169 8002 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:55.021728 4927 rpcz_store.cc:276] 0806 04:07:53.924995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.068715 (+143720us) service_pool.cc:227] Handling call
0806 04:07:54.089117 (+ 20402us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:54.181621 (+ 92504us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:07:54.414439 (+232818us) log.cc:842] Serialized 14 byte log entry
0806 04:07:54.554084 (+139645us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:54.554719 (+ 635us) log.cc:842] Serialized 26 byte log entry
0806 04:07:54.554868 (+ 149us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:07:54.554897 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:54.554964 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:55.015116 (+460152us) raft_consensus.cc:1671] finished
0806 04:07:55.015142 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:55.015288 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":280155,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":1910,"raft.queue_time_us":12406,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":258,"thread_start_us":334261,"threads_started":2,"wal-append.queue_time_us":344911}
I20260806 04:07:55.039273 7754 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:55.004446 7603 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:55.024258 7762 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.025261 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.040650 5026 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:07:55.040323 7603 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.041235 7754 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:55.040323 7762 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:55.042352 7721 raft_consensus.cc:3032] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:55.042522 7603 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.026351 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.043545 7762 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:55.027052 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:55.031818 8104 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260806 04:07:55.031859 7875 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.045606 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.038280 8002 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:55.025504 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.048228 5035 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:55.042822 7721 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:55.047569 7816 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:55.050909 7721 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.052039 7754 consensus_queue.cc:1046] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:07:55.053295 8084 raft_consensus.cc:3032] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:55.053534 7877 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:55.053767 8084 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:55.053999 7877 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:55.054785 8084 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.056088 5035 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:55.057194 8084 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:55.060508 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 50948.9us
I20260806 04:07:55.052328 7721 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:07:55.064015 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 186014us
I20260806 04:07:55.066519 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.067636 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.068499 5032 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:54.985940 8067 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:07:55.082630 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 43816us
I20260806 04:07:55.079105 4813 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:55.089172 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.092717 4996 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.022828 7852 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:54.975445 8078 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:55.094753 4816 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:07:55.096177 7908 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:07:55.096428 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:55.097188 8085 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.085171 8067 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.087159 7830 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:55.104672 8092 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.518s user 0.010s sys 0.005s
I20260806 04:07:55.107420 5032 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:55.125361 4925 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.136222 4804 raft_consensus.cc:2379] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:55.136739 5863 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:55.142931 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 150942us
W20260806 04:07:55.143343 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54650.7us
W20260806 04:07:55.145977 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 76834us
I20260806 04:07:55.137995 7720 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:55.147008 7720 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:55.147466 7720 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:55.130780 8068 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:07:55.153081 4876 leader_election.cc:334] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.565s (SENT)
I20260806 04:07:55.157503 4925 raft_consensus.cc:2396] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:55.159865 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.167140 8112 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:55.167871 8112 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.176198 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.177013 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.177671 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.179080 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.183425 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.161808 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.185642 5857 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:55.191895 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.192844 4923 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:55.196786 5863 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:55.197656 4996 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.170397 8112 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.198870 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:55.141954 7932 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260806 04:07:55.203513 4814 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:07:55.204890 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51843.6us
W20260806 04:07:55.210832 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 88022.1us
I20260806 04:07:55.218209 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.223155 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.224023 5857 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20260806 04:07:55.224455 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.225004 4924 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:55.225479 8085 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:55.227015 8078 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:55.227509 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.228214 4928 raft_consensus.cc:2379] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:55.228950 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:55.230947 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 70418.3us
I20260806 04:07:55.231043 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.231362 4920 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:07:55.232482 4930 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:55.231209 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.233626 5857 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:55.235220 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:55.236243 7830 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:55.236784 8090 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:55.216646 8068 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:55.252995 8108 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:55.253603 8108 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:55.254074 8108 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.254855 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:55.255661 4816 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.257244 4924 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:55.255015 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 95987.2us
I20260806 04:07:55.265398 4924 raft_consensus.cc:1226] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:55.267529 4923 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:55.268179 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.238251 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.266268 4770 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:55.271122 5863 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:07:55.273027 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.274282 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.274554 8078 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:55.289618 4926 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:55.289728 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:55.274964 8124 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260806 04:07:55.275298 7587 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:55.277141 4927 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:55.281638 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.283967 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.287532 5035 raft_consensus.cc:1226] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:07:55.293221 5030 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:55.297096 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.297623 4930 raft_consensus.cc:1261] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 04:07:55.297494 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.297827 5037 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:07:55.298444 5029 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:55.298488 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.299281 5026 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:55.300491 7852 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:55.299799 4996 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.301753 7852 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:55.297116 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.303722 5038 raft_consensus.cc:2379] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:55.303795 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.304491 5034 raft_consensus.cc:2362] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4. Re-sending same reply.
I20260806 04:07:55.302884 8067 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.304131 4996 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.307173 7852 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:55.307401 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.308002 4996 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.308806 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.308908 7852 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:55.309469 5028 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:55.307169 8067 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:55.310428 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 72330.7us
I20260806 04:07:55.291971 7643 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260806 04:07:55.313968 5812 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.311084 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.315574 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.316176 4920 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:55.316627 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.298024 4930 pending_rounds.cc:76] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Aborting all ops after (but not including) 5
I20260806 04:07:55.318940 4930 pending_rounds.cc:97] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 04:07:55.293684 7936 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.319296 4930 raft_consensus.cc:2866] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:55.296584 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.290794 8120 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:55.295185 7587 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:55.308099 5036 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:07:55.312580 4876 leader_election.cc:334] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.607s (SENT)
I20260806 04:07:55.346895 4814 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:07:55.320717 7587 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:55.286872 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:55.362067 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 150101us
I20260806 04:07:55.311239 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:55.369907 4768 connection.cc:232] RPC call timeout handler was delayed by -2.2064122438896447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
I20260806 04:07:55.374773 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.371567 5816 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:55.381338 7994 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.382695 7721 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:07:55.383247 7721 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:55.381935 7994 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.384219 7721 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.385604 4926 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:55.386920 7994 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:55.389693 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 77142.2us
W20260806 04:07:55.390451 4876 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.657s (SENT)
I20260806 04:07:55.390508 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.324710 7936 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.391316 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.393297 5026 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:55.393668 5035 raft_consensus.cc:2379] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:55.354314 8090 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:55.409192 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:55.410256 5036 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:55.422101 4992 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.418010 4818 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:55.394992 4923 raft_consensus.cc:2396] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 04:07:55.394575 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.400065 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:55.402858 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 91710.3us
I20260806 04:07:55.431102 5028 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:55.405187 4920 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:55.368300 8084 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:55.439474 4992 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:55.392197 7876 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:55.399178 7852 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:55.449873 4925 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:55.450393 4924 raft_consensus.cc:2379] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:55.452479 4768 connection.cc:232] RPC call timeout handler was delayed by -2.0381122438702732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2235s for a response
I20260806 04:07:55.453572 4813 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:55.360277 7720 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.456161 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.459407 5036 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.451826 7816 raft_consensus.cc:2781] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:55.462635 5853 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 04:07:55.463068 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4087423819582909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
W20260806 04:07:55.463593 4768 connection.cc:232] RPC call timeout handler was delayed by -1.3934423821046948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
I20260806 04:07:55.463120 7816 raft_consensus.cc:686] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:55.450420 8069 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
W20260806 04:07:55.463948 4768 connection.cc:232] RPC call timeout handler was delayed by -1.3241423820145428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
I20260806 04:07:55.436486 8084 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:55.475400 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 156890us
I20260806 04:07:55.478633 4816 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:55.484726 4996 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.487380 5816 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:55.455379 7830 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:55.452884 7852 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:55.493530 7852 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:55.492465 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0355122438631952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33490000000000003s for a response
W20260806 04:07:55.494676 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 104253us
W20260806 04:07:55.494753 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0188768720254302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260806 04:07:55.495796 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0004122438840568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3388s for a response
W20260806 04:07:55.495995 4876 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.711s (SENT)
I20260806 04:07:55.503382 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:55.486881 8084 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:55.518925 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:07:55.524251 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 59016.4us
I20260806 04:07:55.523651 5036 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:07:55.498991 7875 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.527051 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.484054 7603 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 04:07:55.543505 4996 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:55.545130 4876 leader_election.cc:334] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.046s (SENT)
I20260806 04:07:55.501343 7762 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:55.515897 7816 consensus_queue.cc:237] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.535617 8141 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260806 04:07:55.563645 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.544382 4927 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:55.551090 4924 raft_consensus.cc:2379] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:55.555037 8069 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:07:55.559113 8127 log.cc:924] Time spent T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260806 04:07:55.559693 7720 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.559484 7875 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.596473 4927 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:55.597630 4927 raft_consensus.cc:595] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:55.598141 4927 raft_consensus.cc:603] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:55.598694 7875 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.602151 8116 raft_consensus.cc:2741] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
W20260806 04:07:55.602414 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 83583.7us
I20260806 04:07:55.561477 5816 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:55.567315 4925 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:55.570036 7603 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:55.617658 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:55.617990 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 73434.4us
I20260806 04:07:55.525827 7754 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:55.579993 8068 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:55.579916 7852 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:55.580988 8063 log.cc:924] Time spent T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260806 04:07:55.582013 8128 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 0.135s user 0.002s sys 0.000s
I20260806 04:07:55.521505 7877 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.583379 5812 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:55.574707 8135 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260806 04:07:55.618799 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 57420.1us
W20260806 04:07:55.645293 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 92401.1us
W20260806 04:07:55.636847 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 133579us
I20260806 04:07:55.578311 8108 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.649009 4992 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:55.630821 8067 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:55.631433 7603 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:07:55.658021 4925 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da2e93c822734c02a26703dabb801932"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:07:55.665346 4925 raft_consensus.cc:595] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:55.665673 4925 raft_consensus.cc:603] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:55.642879 7754 raft_consensus.cc:686] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:55.654551 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:55.667960 4992 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
W20260806 04:07:55.683104 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 117391us
I20260806 04:07:55.647181 8120 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:55.684216 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.685561 4992 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.632004 7587 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.690409 5812 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:55.691756 8141 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:55.692420 8141 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:55.689945 7603 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:55.693473 7603 raft_consensus.cc:686] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:55.692965 8141 raft_consensus.cc:686] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:55.692171 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:55.699513 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 96161.7us
I20260806 04:07:55.661110 8067 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:55.731114 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:55.673713 8116 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:07:55.733726 4876 leader_election.cc:334] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.196s (SENT)
W20260806 04:07:55.735584 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 172016us
I20260806 04:07:55.686025 8068 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:55.695950 7932 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:55.692188 7587 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.732223 7908 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.737113 8116 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:55.731547 8067 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:55.751139 4876 leader_election.cc:334] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.205s (SENT)
I20260806 04:07:55.757504 4876 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:55.751165 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 59115.5us
W20260806 04:07:55.751905 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 66417.4us
I20260806 04:07:55.738484 7643 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:55.749029 7830 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:07:55.736346 8090 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.760696 4992 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:07:55.755594 8120 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:55.759778 8116 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:55.776835 7754 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.765575 7643 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:55.754866 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:55.782533 8120 raft_consensus.cc:686] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:55.765870 8158 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:55.798343 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.798709 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:55.799057 5032 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:55.800172 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.802079 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:55.758589 7908 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.783622 7877 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:55.784011 8156 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:55.784384 7852 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:55.750144 7603 consensus_queue.cc:237] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:55.786322 8084 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.786968 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:55.832424 4805 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:55.832650 5862 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:55.791246 4808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:55606
I20260806 04:07:55.833982 4808 raft_consensus.cc:595] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:55.834335 4808 raft_consensus.cc:603] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:07:55.792476 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.794349 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:55.835173 5026 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:55.797732 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.797842 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.835986 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.836390 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:55.836607 5028 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:07:55.837008 5032 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:55.804044 4768 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:55.773488 8094 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.798209 8120 consensus_queue.cc:237] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.753235 7932 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.828794 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.909050 5866 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 7: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:55.830206 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.787485 8141 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.862195 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.845686 5036 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 6 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:07:55.808501 7888 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:55.910223 5860 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:55.776140 8090 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.845489 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.791029 7830 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260806 04:07:55.866829 8108 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:55.805330 7643 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:55.795955 7994 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:55.864051 5867 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:55.868496 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.903528 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:55.907636 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:55.854339 8156 raft_consensus.cc:686] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:55.855199 7908 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:55.913736 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:55.915830 4879 leader_election.cc:334] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Timed out: RequestConsensusVote RPC to 127.2.248.68:38873 timed out after 1.621s (SENT)
I20260806 04:07:55.917670 4810 raft_consensus.cc:2354] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:07:55.918620 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:55.931128 4931 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:55.862174 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.924022 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:55.902238 8087 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:55.901587 8068 raft_consensus.cc:686] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:55.934396 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:55.935874 5032 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:55.945613 4925 raft_consensus.cc:2379] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:55.939317 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:55.955638 5812 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:55.925604 7888 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:55.960357 5866 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 7: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:55.959916 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:55.909291 8094 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:55.962535 5859 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
W20260806 04:07:55.970994 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 62752.7us
I20260806 04:07:55.977999 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:55.946683 8162 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:07:55.988209 4876 leader_election.cc:334] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.608s (SENT)
I20260806 04:07:55.989692 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:55.992496 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.001060 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:56.009712 4818 raft_consensus.cc:2396] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:07:55.984431 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:56.021835 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.024883 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.025772 4879 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:56.022617 4804 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:56.015060 8156 consensus_queue.cc:237] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.029393 8084 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:56.034878 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 127449us
I20260806 04:07:56.034983 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:56.021487 8155 log.cc:924] Time spent T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.246s user 0.001s sys 0.000s
I20260806 04:07:55.984225 7721 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:07:56.038777 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:55.956562 8087 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:56.048094 4876 leader_election.cc:334] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.752s (SENT)
I20260806 04:07:56.065671 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.066440 5035 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:56.062597 5866 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:56.048432 4879 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:56.050436 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.094286 4879 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:56.097150 5032 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:56.105682 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 57619.8us
I20260806 04:07:56.067808 8085 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:07:56.115262 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.121999 4879 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:56.123982 4879 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:56.042356 7936 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 04:07:56.024091 8162 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.139673 5860 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:56.117017 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 04:07:56.144912 5035 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
W20260806 04:07:56.147019 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 343082us
I20260806 04:07:56.096172 8177 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:56.145138 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:07:56.161718 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:56.165496 5032 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:56.115465 7994 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:07:56.097589 7888 consensus_queue.cc:1046] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260806 04:07:56.057515 8068 consensus_queue.cc:237] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:56.233297 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 201786us
I20260806 04:07:56.134070 7932 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.148077 8090 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.150455 7979 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:07:56.260597 4817 consensus_meta.cc:287] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.585s user 0.010s sys 0.005s
I20260806 04:07:56.153860 8177 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:56.267218 4817 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:56.278471 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:56.280798 5028 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:07:56.265803 8094 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.285904 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:07:56.287318 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 157984us
I20260806 04:07:56.293538 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.286803 4808 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:07:56.278365 8177 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:56.294991 8142 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:56.320645 4768 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:07:56.333176 8067 consensus_meta.cc:287] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.562s user 0.011s sys 0.000s
I20260806 04:07:56.323117 8087 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.333688 8067 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:56.320083 8173 log.cc:924] Time spent T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.286s user 0.001s sys 0.000s
I20260806 04:07:56.337205 5863 raft_consensus.cc:2379] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:56.330473 4805 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:56.340746 5862 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:56.349351 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.354192 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.355402 4813 raft_consensus.cc:2396] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 04:07:56.355593 5854 consensus_meta.cc:287] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.522s user 0.012s sys 0.000s
I20260806 04:07:56.356637 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:56.359467 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 43531.5us
I20260806 04:07:56.365437 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.365617 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.375380 4808 raft_consensus.cc:2354] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:56.380509 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.353132 8087 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:07:56.351351 8067 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:56.403432 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 137079us
I20260806 04:07:56.407260 5812 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:56.411167 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.411157 5812 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:56.356091 5854 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:56.378808 7603 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:56.416851 8084 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:07:56.363692 7762 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260806 04:07:56.405248 8087 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:56.437347 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.424118 8067 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:07:56.433947 4805 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:07:56.423286 8140 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:56.453383 4813 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:07:56.457247 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.458472 4805 raft_consensus.cc:3037] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:56.429450 8162 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.444773 8087 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:07:56.428678 7603 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.469666 7816 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.470399 7816 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.473058 7816 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:07:56.474300 8188 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 04:07:56.476019 7603 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:56.479838 7720 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:07:56.480345 7720 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:56.487566 4768 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:56.484583 7994 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:56.491875 8183 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260806 04:07:56.491467 7994 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.495541 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:56.498168 7932 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.457926 8067 raft_consensus.cc:686] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:56.499764 8067 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.461863 8069 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:56.503302 8069 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.504333 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:56.494364 7994 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.515129 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:56.517135 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 106087us
W20260806 04:07:56.518196 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 222027us
I20260806 04:07:56.518427 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:56.521615 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.511003 8069 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:56.539570 5812 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:56.539115 4805 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:07:56.541229 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.541909 4816 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:56.526750 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:56.544993 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.545699 5028 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:56.512174 7889 raft_consensus.cc:2741] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260806 04:07:56.546343 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.547979 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:56.546729 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 79401.3us
I20260806 04:07:56.553051 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.481343 7720 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.553700 5032 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:07:56.564878 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 74229.2us
I20260806 04:07:56.528565 7721 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:56.574497 4805 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:56.509681 7932 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.590983 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.591777 5032 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:56.604425 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 77767.8us
I20260806 04:07:56.610319 4816 raft_consensus.cc:2396] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:07:56.613463 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.571798 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:56.618889 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:56.627120 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 61030.6us
I20260806 04:07:56.629923 4816 raft_consensus.cc:2379] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:56.614125 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.617406 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:56.613667 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.631623 4924 raft_consensus.cc:2379] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:07:56.608647 8208 log.cc:924] Time spent T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260806 04:07:56.633698 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 68690.2us
I20260806 04:07:56.609503 5857 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:56.613034 8142 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:56.663055 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.663802 5035 raft_consensus.cc:2362] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:56.665321 7932 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.628623 7720 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:56.613404 7721 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:56.666318 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.667790 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.671234 5034 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:56.671730 5034 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:56.614755 8160 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.634456 7936 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:07:56.674633 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.674216 5028 raft_consensus.cc:1261] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:07:56.665210 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:56.680342 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.683094 5037 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:56.683241 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.683940 5035 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:56.684790 4879 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:56.684894 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:07:56.685775 8201 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20260806 04:07:56.685858 5026 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
W20260806 04:07:56.691557 8205 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.074s user 0.000s sys 0.002s
I20260806 04:07:56.695945 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.698534 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.668524 7419 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.704644 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.709801 5034 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:56.686933 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.708617 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.713435 5029 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:56.713858 5029 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:56.719426 8192 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.720341 8192 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.724422 8192 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.728082 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.730459 5038 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:07:56.732066 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 47396.6us
I20260806 04:07:56.735576 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.728597 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1475) took 1080 ms. Trace:
I20260806 04:07:56.737232 4927 rpcz_store.cc:276] 0806 04:07:55.648266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:55.648424 (+ 158us) service_pool.cc:227] Handling call
0806 04:07:55.648769 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:55.662001 (+ 13232us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:55.662045 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:55.684779 (+ 22734us) log.cc:842] Serialized 26 byte log entry
0806 04:07:56.008922 (+324143us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b1800370b40. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a14345 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e1411 00007f23d992e5cf
0806 04:07:56.258478 (+249556us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:07:56.430257 (+171779us) log.cc:842] Serialized 14 byte log entry
0806 04:07:56.704878 (+274621us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:56.704957 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:56.705000 (+ 43us) raft_consensus.cc:1671] finished
0806 04:07:56.728342 (+ 23342us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:56.728512 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":194147,"prepare.queue_time_us":72545,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":179820,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":13451,"spinlock_wait_cycles":4456832,"thread_start_us":508588,"threads_started":3,"wal-append.queue_time_us":653433}
I20260806 04:07:56.742050 5037 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:07:56.674847 7714 consensus_queue.cc:1035] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260806 04:07:56.704470 7889 raft_consensus.cc:2741] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260806 04:07:56.744087 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.679026 8068 consensus_queue.cc:1046] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260806 04:07:56.740665 8113 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:56.712595 7936 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:56.753247 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.756193 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.758870 4817 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:56.759013 5038 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:56.742844 7932 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:07:56.759882 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 109819us
W20260806 04:07:56.761596 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 96476.7us
I20260806 04:07:56.762344 4879 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:56.750563 5030 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
W20260806 04:07:56.741547 7643 consensus_meta.cc:287] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.799s user 0.012s sys 0.002s
I20260806 04:07:56.764194 5028 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:07:56.751579 8219 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:56.730034 7419 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.769291 5028 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:56.769506 5037 raft_consensus.cc:1261] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:56.681321 8160 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.771203 5035 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:56.773690 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.780694 8229 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:07:56.767773 7643 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.787899 7643 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.789539 5029 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:56.793479 8160 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.794950 4882 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:56.773095 8116 consensus_meta.cc:287] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.981s user 0.014s sys 0.001s
I20260806 04:07:56.797493 4805 raft_consensus.cc:1226] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:56.797864 5038 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:56.797617 4770 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:56.798887 5030 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:56.801054 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.801581 4814 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:07:56.802016 4816 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:56.765988 8224 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:56.800441 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:56.816123 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.817396 5029 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:56.818601 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:56.747411 7721 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.826910 4810 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:07:56.822741 7875 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:07:56.825870 8094 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 04:07:56.832795 8192 consensus_queue.cc:1035] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 04:07:56.827170 8002 raft_consensus.cc:2781] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:56.824728 7936 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.876258 8002 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:56.876730 8002 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:56.839870 8200 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.877621 8200 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.800967 8116 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.879832 7419 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.880542 8116 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.776510 8219 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.882226 4804 raft_consensus.cc:1226] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:56.883266 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:56.883778 8219 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:56.868276 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:56.887394 7875 raft_consensus.cc:686] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:07:56.888554 8177 consensus_meta.cc:287] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.565s user 0.011s sys 0.000s
I20260806 04:07:56.888754 7875 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.889081 8177 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.890442 4927 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 04:07:56.891274 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 59586.1us
I20260806 04:07:56.937361 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.935352 8215 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:56.942057 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:56.941787 8215 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:56.943324 8171 raft_consensus.cc:2741] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260806 04:07:56.943730 4882 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:56.944655 7419 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:56.946153 5036 raft_consensus.cc:1261] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:56.949448 8215 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.951210 8087 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:56.952867 8177 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.953336 8087 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:56.919247 8200 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:56.954682 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:56.955341 4927 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:56.955293 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:56.956043 5862 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:56.938359 4808 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:56.960234 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:56.960839 5034 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:56.940850 8229 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260806 04:07:56.854547 8230 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.197s
I20260806 04:07:56.973687 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:56.974355 4817 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:56.958005 4882 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:56.976985 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 48959.6us
I20260806 04:07:56.977519 5862 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:56.980042 5037 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:56.981495 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:56.982285 5037 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:57.019292 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:57.022004 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:57.023000 4809 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:07:57.038676 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 260731us
I20260806 04:07:57.038936 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.039837 5036 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:07:57.041128 5034 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:07:57.045902 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:07:57.042385 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 56010.8us
I20260806 04:07:57.046880 5036 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:57.048262 4808 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:57.049160 4818 raft_consensus.cc:1261] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260806 04:07:57.049633 4818 pending_rounds.cc:76] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 1
I20260806 04:07:57.049903 4818 pending_rounds.cc:97] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 04:07:57.050181 4818 raft_consensus.cc:2866] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:57.059240 4809 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260806 04:07:57.062449 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:57.048204 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:57.051492 4816 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:57.066092 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.066939 5037 raft_consensus.cc:2362] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5. Re-sending same reply.
I20260806 04:07:57.075666 4817 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:57.077288 4882 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:57.079000 8180 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:07:57.080063 8180 raft_consensus.cc:686] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:57.081441 8180 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.059657 4809 pending_rounds.cc:76] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 1
I20260806 04:07:57.085140 4809 pending_rounds.cc:97] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 04:07:57.085418 4809 raft_consensus.cc:2866] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:57.086633 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.087739 5034 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 3: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:57.042006 7419 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.042975 8215 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:57.108363 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:57.059849 8229 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.119102 8232 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.081s
I20260806 04:07:57.119210 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.129146 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.129792 5037 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:57.129997 5034 raft_consensus.cc:2362] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:57.120795 8235 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:07:57.122273 8158 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:57.107932 8226 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260806 04:07:57.104928 5036 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:57.127460 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:57.085597 8177 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:57.051492 8087 consensus_queue.cc:1046] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260806 04:07:57.141456 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.156210 5034 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:57.156204 8087 raft_consensus.cc:3032] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:57.144351 8156 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:57.157467 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.157444 8158 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.158383 5034 raft_consensus.cc:2362] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:07:57.159462 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.045996 8002 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.160638 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.161441 5034 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:57.161835 5034 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:57.162670 8002 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:57.163825 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:57.163987 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.164435 4809 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:57.164876 5034 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:57.165637 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.165773 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:57.065647 8192 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260806 04:07:57.107430 8215 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:57.166985 8215 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:57.112253 7419 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.151166 7852 raft_consensus.cc:3032] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:07:57.137588 8229 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.134279 8235 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:57.156883 8087 raft_consensus.cc:729] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:57.157411 8156 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:57.148613 8236 log.cc:924] Time spent T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260806 04:07:57.166201 5034 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:57.191356 5034 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:07:57.144685 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.172541 4809 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:07:57.183172 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 55458.1us
I20260806 04:07:57.105511 7932 raft_consensus.cc:686] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:57.160571 8158 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:57.179275 8219 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.187446 7936 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:07:57.211181 7419 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:57.211393 5034 raft_consensus.cc:1261] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:57.212231 7932 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.181348 7852 raft_consensus.cc:729] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:07:57.186810 8235 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:57.231366 4882 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:57.225921 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:57.238070 4768 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:57.232622 8219 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.235409 5034 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:07:57.239490 5036 raft_consensus.cc:1226] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:57.181761 8160 consensus_queue.cc:1035] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260806 04:07:57.181478 8002 consensus_queue.cc:1046] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.040s
I20260806 04:07:57.241927 5859 raft_consensus.cc:1226] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260806 04:07:57.240379 5037 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:57.246589 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:57.247545 4809 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:57.185842 8177 consensus_queue.cc:1035] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 04:07:57.264338 4882 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:57.265659 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 134189us
I20260806 04:07:57.243391 7936 raft_consensus.cc:2781] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:07:57.272831 5866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 282) took 1314 ms. Trace:
I20260806 04:07:57.273705 5866 rpcz_store.cc:276] 0806 04:07:55.955234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:55.963896 (+ 8662us) service_pool.cc:227] Handling call
0806 04:07:57.269797 (+1305901us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b54004fced4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:57.269914 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192243072}
I20260806 04:07:57.274706 4768 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:57.261763 8116 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:07:57.277460 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 92222.9us
I20260806 04:07:57.262918 8068 raft_consensus.cc:465] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Not starting pre-election -- already a leader
I20260806 04:07:57.263443 8192 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:07:57.276140 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:57.286020 4996 leader_election.cc:398] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
W20260806 04:07:57.276569 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 56396.4us
I20260806 04:07:57.290002 4996 leader_election.cc:401] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:07:57.289708 8192 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:57.291283 8192 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:57.291940 4996 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.294555 5859 raft_consensus.cc:2379] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:07:57.299250 4996 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.299702 8090 raft_consensus.cc:3032] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:57.300554 4879 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.277483 7643 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:57.278766 8116 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:57.260326 8087 consensus_queue.cc:260] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.277293 8156 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:57.281606 8002 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:57.310719 8112 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:57.289830 7762 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.300191 8090 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:57.332163 8239 consensus_queue.cc:1046] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:07:57.332628 8192 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.333316 8219 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:57.335712 8255 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:57.324054 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:57.374464 8002 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:57.407158 8255 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:57.407557 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 92981.8us
I20260806 04:07:57.407742 8255 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.408308 7762 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.409314 7643 raft_consensus.cc:2741] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260806 04:07:57.290019 7936 raft_consensus.cc:686] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:57.410770 7936 consensus_queue.cc:237] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.412710 8192 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:57.280992 7852 consensus_queue.cc:260] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.413997 8239 raft_consensus.cc:3032] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:57.405660 8156 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:07:57.363237 8116 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:07:57.414762 8158 consensus_queue.cc:1159] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d7c76b6dcd8945cd818b8107caf4a174" responder_term: 6 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [3.6-5.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:07:57.414615 8239 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:57.417775 8255 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:57.348441 8215 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.338608 8229 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:57.404402 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:57.414747 7852 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:57.419092 7762 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:57.407474 8002 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:57.408728 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:57.423427 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:57.424640 7643 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:57.425179 7643 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:57.425734 7643 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:57.427073 8215 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:57.427623 8235 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.417421 8239 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:57.438561 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 138248us
I20260806 04:07:57.439865 4996 leader_election.cc:398] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:57.440294 4996 leader_election.cc:401] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:07:57.440907 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.442137 8261 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:07:57.446705 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 281023us
I20260806 04:07:57.449702 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:57.450613 5028 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:57.454010 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:57.455073 5862 raft_consensus.cc:2396] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:07:57.455236 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:57.451434 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:57.467041 8219 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:57.468647 4996 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.470907 8275 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:57.471493 7762 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:07:57.471558 8275 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.473464 4996 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.476541 8276 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:57.477643 8276 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:57.479470 8276 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.479807 8219 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:07:57.481958 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:57.476872 8275 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:57.486778 8219 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:07:57.482589 8276 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:57.491209 4996 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:57.498970 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 75715.6us
W20260806 04:07:57.501924 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 47514.4us
I20260806 04:07:57.434478 8192 consensus_queue.cc:1046] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20260806 04:07:57.501680 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 47027.3us
I20260806 04:07:57.448661 8261 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:57.452780 8215 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.064s
I20260806 04:07:57.453609 8090 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.510396 5028 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:57.500159 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:57.504040 8276 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:07:58.888594 8276 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:58.889189 8276 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:57.498605 8239 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:58.892438 8294 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.625703 8267 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:57.541090 8235 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:57.660614 8261 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:58.894951 8267 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:58.897181 8267 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:57.506168 8219 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:58.897810 8219 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.610895 7643 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.538390 7932 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20260806 04:07:57.560185 8002 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.601540 8282 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:59.030675 8282 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:59.031199 8282 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:57.497123 8068 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:07:57.550854 8116 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.541392 8253 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:59.041710 8253 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.042923 8116 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:57.738507 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:07:57.784729 8199 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.324s user 0.003s sys 0.005s
I20260806 04:07:57.544167 5862 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:07:59.044253 8253 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.044751 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1477) took 3027 ms. Trace:
I20260806 04:07:59.044886 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 290) took 1501 ms. Trace:
I20260806 04:07:59.045302 5862 rpcz_store.cc:276] 0806 04:07:57.543121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.543259 (+ 138us) service_pool.cc:227] Handling call
0806 04:07:57.543728 (+ 469us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.044613 (+1500885us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.044830 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:59.046584 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:59.045207 4925 rpcz_store.cc:276] 0806 04:07:56.017571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.017781 (+ 210us) service_pool.cc:227] Handling call
0806 04:07:56.043984 (+ 26203us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:56.070013 (+ 26029us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:56.070062 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:56.091229 (+ 21167us) log.cc:842] Serialized 26 byte log entry
0806 04:07:56.316908 (+225679us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:56.316946 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:56.317024 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:57.783844 (+1466820us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:59.044393 (+1260549us) raft_consensus.cc:1671] finished
0806 04:07:59.044420 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:59.044660 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12627,"raft.queue_time_us":14731,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":18545,"spinlock_wait_cycles":15836928,"thread_start_us":173439,"threads_started":1,"wal-append.queue_time_us":309169}
I20260806 04:07:59.047271 4817 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:59.047796 8112 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:57.885035 8281 raft_consensus.cc:3032] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:59.049355 8281 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:07:57.871929 5853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 551) took 1359 ms. Trace:
I20260806 04:07:59.050253 8281 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.660718 8090 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:59.051303 8002 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.051409 8281 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:59.054196 4668 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46900 (request call id 281) took 13669 ms (client timeout 15000 ms). Trace:
I20260806 04:07:57.597539 5863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 270) took 1801 ms. Trace:
I20260806 04:07:57.923729 8226 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.543327 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:59.055159 4805 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:59.055068 8226 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.050813 5853 rpcz_store.cc:276] 0806 04:07:56.512226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.514373 (+ 2147us) service_pool.cc:227] Handling call
0806 04:07:57.604590 (+1090217us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:57.604845 (+ 255us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:57.605401 (+ 556us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:57.605442 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:57.606101 (+ 659us) log.cc:842] Serialized 26 byte log entry
0806 04:07:57.871291 (+265190us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:57.871361 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:57.871474 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:57.871538 (+ 64us) raft_consensus.cc:1671] finished
0806 04:07:57.871555 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:57.871839 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1473,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":147,"spinlock_wait_cycles":8094080,"thread_start_us":264677,"threads_started":1,"wal-append.queue_time_us":64046,"wal-append.run_cpu_time_us":2163,"wal-append.run_wall_time_us":1002349}
I20260806 04:07:57.961294 4813 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:07:59.057384 8226 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.057780 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2119) took 1098 ms. Trace:
I20260806 04:07:57.962384 4809 raft_consensus.cc:1261] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:07:59.058948 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.059229 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2121) took 1099 ms. Trace:
I20260806 04:07:59.059644 4814 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:59.059571 4809 rpcz_store.cc:276] 0806 04:07:57.959887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.961743 (+ 1856us) service_pool.cc:227] Handling call
0806 04:07:57.962044 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.058967 (+1096923us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.059176 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:57.966603 4808 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:07:59.060585 8297 raft_consensus.cc:2741] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260806 04:07:59.060926 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2120) took 1101 ms. Trace:
I20260806 04:07:59.061070 4882 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.061324 4808 rpcz_store.cc:276] 0806 04:07:57.959613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.965909 (+ 6296us) service_pool.cc:227] Handling call
0806 04:07:57.966408 (+ 499us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.060676 (+1094268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.060870 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66560}
I20260806 04:07:59.062109 8297 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.054836 5863 rpcz_store.cc:276] 0806 04:07:55.796301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:55.796783 (+ 482us) service_pool.cc:227] Handling call
0806 04:07:57.597238 (+1800455us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54006ed054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:57.597443 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":980718336}
I20260806 04:07:59.062595 8297 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.063228 8112 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:59.063661 8112 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:59.058164 4813 rpcz_store.cc:276] 0806 04:07:57.959274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.960529 (+ 1255us) service_pool.cc:227] Handling call
0806 04:07:57.960917 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.057490 (+1096573us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.057722 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:59.063998 8112 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:57.976758 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2094) took 1923 ms. Trace:
I20260806 04:07:57.992241 7875 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:57.537930 4876 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.065418 4804 rpcz_store.cc:276] 0806 04:07:56.053669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.054378 (+ 709us) service_pool.cc:227] Handling call
0806 04:07:56.054832 (+ 454us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:56.882678 (+827846us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:57.976440 (+1093762us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54000ffd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:57.976679 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1902333184}
W20260806 04:07:59.066133 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.52819e+06us
I20260806 04:07:59.065671 7875 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:58.324772 8192 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:59.066581 8297 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.066969 8192 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.068107 7875 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.068552 8242 consensus_queue.cc:1035] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:59.069063 8192 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:58.365522 8283 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:58.670240 4720 debug-util.cc:398] Leaking SignalData structure 0x7b080003e460 after lost signal to thread 3045
I20260806 04:07:59.069821 8283 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:59.070672 4720 debug-util.cc:398] Leaking SignalData structure 0x7b0800355bc0 after lost signal to thread 4724
I20260806 04:07:59.070389 8242 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:59.071209 8242 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:59.071624 8242 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:07:59.072137 4720 debug-util.cc:398] Leaking SignalData structure 0x7b08001670e0 after lost signal to thread 4853
W20260806 04:07:57.518810 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 229633us
I20260806 04:07:57.527704 5859 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260806 04:07:59.073719 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 289) took 1548 ms. Trace:
I20260806 04:07:59.073938 5816 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:59.074096 5859 rpcz_store.cc:276] 0806 04:07:57.525151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.525344 (+ 193us) service_pool.cc:227] Handling call
0806 04:07:57.525699 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.073426 (+1547727us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.073661 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:58.745867 8291 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.076972 8226 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:07:59.077309 8291 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.078526 8226 consensus_queue.cc:1046] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:07:59.079130 8200 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:07:59.079699 8291 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:58.784125 8292 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:59.079998 7994 raft_consensus.cc:3032] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:07:59.080470 7994 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:59.080502 8292 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.081465 7994 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:57.624158 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 126248us
I20260806 04:07:59.083104 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.085064 8283 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:58.819795 8230 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.086956 8230 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.088014 5854 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:59.089128 8230 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:58.876133 5028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 238) took 1427 ms (client timeout 1684 ms). Trace:
I20260806 04:07:58.892973 8294 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:59.089706 5028 rpcz_store.cc:269] 0806 04:07:57.449093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.449363 (+ 270us) service_pool.cc:227] Handling call
0806 04:07:58.876006 (+1426643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:58.920310 8219 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:58.923563 8261 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.120604 8108 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:59.121196 8108 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:59.122074 8261 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:57.524112 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:58.926956 4818 raft_consensus.cc:1226] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:59.123082 5866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 561) took 1619 ms. Trace:
I20260806 04:07:59.123449 5866 rpcz_store.cc:276] 0806 04:07:57.503465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.515969 (+ 12504us) service_pool.cc:227] Handling call
0806 04:07:59.123023 (+1607054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:58.926612 7643 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:58.941325 8295 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:58.945808 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1953) took 2010 ms. Trace:
W20260806 04:07:58.949239 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.44901e+06us
I20260806 04:07:59.124922 5036 rpcz_store.cc:276] 0806 04:07:56.935347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.181311 (+245964us) service_pool.cc:227] Handling call
0806 04:07:57.181645 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:57.239909 (+ 58264us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:58.945468 (+1705559us) spinlock_profiling.cc:243] Waited 57.6 ms on lock 0x7b54002d28d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:58.945721 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":132467584}
I20260806 04:07:58.951045 8140 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:59.126137 8140 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:59.122107 8108 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:59.054623 4668 rpcz_store.cc:269] 0806 04:07:45.380838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:45.391198 (+ 10360us) service_pool.cc:227] Handling call
0806 04:07:59.050617 (+13659419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:54.716971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:54.909525 (+192554us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:54.909581 (+ 56us) write_op.cc:435] Acquired schema lock
0806 04:07:54.909593 (+ 12us) tablet.cc:660] Decoding operations
0806 04:07:54.910354 (+ 761us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:07:54.910443 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0806 04:07:54.910472 (+ 29us) tablet.cc:683] Acquiring locks for 5 operations
0806 04:07:54.976278 (+ 65806us) tablet.cc:699] Row locks acquired
0806 04:07:55.001388 (+ 25110us) write_op.cc:260] PREPARE: finished
0806 04:07:55.057082 (+ 55694us) write_op.cc:270] Start()
0806 04:07:55.057303 (+ 221us) write_op.cc:276] Timestamp: P: 1785989275021991 usec, L: 0
0806 04:07:55.057352 (+ 49us) op_driver.cc:331] REPLICATION: starting
0806 04:07:55.121686 (+ 64334us) log.cc:842] Serialized 2079 byte log entry
0806 04:07:56.254305 (+1132619us) op_driver.cc:444] REPLICATION: finished
0806 04:07:56.572147 (+317842us) write_op.cc:301] APPLY: starting
0806 04:07:56.608408 (+ 36261us) tablet.cc:1365] starting BulkCheckPresence
0806 04:07:56.669993 (+ 61585us) tablet.cc:1368] finished BulkCheckPresence
0806 04:07:56.729280 (+ 59287us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:07:56.873424 (+144144us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:07:56.873886 (+ 462us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0806 04:07:56.873966 (+ 80us) write_op.cc:312] APPLY: finished
0806 04:07:56.874413 (+ 447us) log.cc:842] Serialized 47 byte log entry
0806 04:07:56.874734 (+ 321us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:56.874876 (+ 142us) write_op.cc:454] Released schema lock
0806 04:07:56.874958 (+ 82us) write_op.cc:341] FINISH: Updating metrics
0806 04:07:56.875444 (+ 486us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:07:56.875486 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":244517,"apply.run_cpu_time_us":5510,"apply.run_wall_time_us":329499,"num_ops":5,"prepare.queue_time_us":409795,"prepare.run_cpu_time_us":25645,"prepare.run_wall_time_us":678534,"raft.queue_time_us":367529,"raft.run_cpu_time_us":8449,"raft.run_wall_time_us":282895,"replication_time_us":1116043,"thread_start_us":629931,"threads_started":4,"wal-append.queue_time_us":291950,"wal-append.run_cpu_time_us":49317,"wal-append.run_wall_time_us":3341894}]]}
I20260806 04:07:58.953840 8276 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:58.956625 8302 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:58.960664 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:58.957826 8304 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:58.982342 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:58.983037 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.141063 5030 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:58.975018 5860 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:07:58.983633 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1940) took 2241 ms. Trace:
I20260806 04:07:58.980438 8212 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:58.974021 8087 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:58.978799 8180 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:58.976210 8172 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:58.998236 7419 raft_consensus.cc:3032] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:07:59.000308 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 231) took 2201 ms. Trace:
I20260806 04:07:59.157899 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.012157 8306 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:59.015651 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.019096 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.040385 8299 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.171183 5029 raft_consensus.cc:2354] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6.
I20260806 04:07:59.171862 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1947) took 2347 ms. Trace:
I20260806 04:07:59.172266 5029 rpcz_store.cc:276] 0806 04:07:56.824912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.825049 (+ 137us) service_pool.cc:227] Handling call
0806 04:07:59.018947 (+2193898us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:59.171778 (+152831us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6668032}
W20260806 04:07:57.852003 8251 log.cc:924] Time spent T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.560s user 0.004s sys 0.000s
I20260806 04:07:59.165279 7419 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:59.153232 8304 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.183843 7419 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.185168 7419 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.185346 8304 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.167193 8180 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.170024 8172 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.188557 8180 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:57.919629 8252 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.628s user 0.002s sys 0.000s
I20260806 04:07:57.972048 5038 raft_consensus.cc:1226] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:07:59.072525 8282 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.086390 4996 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.090559 5854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 634) took 2286 ms. Trace:
I20260806 04:07:59.120042 8294 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.119732 8067 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 04:07:59.125411 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:59.212868 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 87531.5us
I20260806 04:07:59.124954 8295 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:57.519426 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.092509 5028 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260806 04:07:59.095964 4720 debug-util.cc:398] Leaking SignalData structure 0x7b08003499a0 after lost signal to thread 4966
W20260806 04:07:59.130123 4876 connection.cc:232] RPC call timeout handler was delayed by -2.5382252091076225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20260806 04:07:59.221045 4876 leader_election.cc:334] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.299s (SENT)
I20260806 04:07:59.221594 4876 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:59.221859 4720 debug-util.cc:398] Leaking SignalData structure 0x7b0800252220 after lost signal to thread 5070
I20260806 04:07:59.110606 4817 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
W20260806 04:07:59.222359 4720 debug-util.cc:398] Leaking SignalData structure 0x7b080037ff60 after lost signal to thread 5071
I20260806 04:07:59.091602 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:07:59.223511 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 132010us
W20260806 04:07:59.224753 4720 debug-util.cc:398] Leaking SignalData structure 0x7b0800375600 after lost signal to thread 5901
I20260806 04:07:59.132355 4805 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
W20260806 04:07:59.227007 4805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 2118) took 1684 ms (client timeout 1906 ms). Trace:
W20260806 04:07:59.227501 4805 rpcz_store.cc:269] 0806 04:07:57.542690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.543020 (+ 330us) service_pool.cc:227] Handling call
0806 04:07:59.226881 (+1683861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:59.126529 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.228905 5036 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:07:59.092393 4669 catalog_manager.cc:5495] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 4, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 4 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 04:07:59.230697 4720 debug-util.cc:398] Leaking SignalData structure 0x7b0800375640 after lost signal to thread 5902
I20260806 04:07:59.231664 4669 catalog_manager.cc:5495] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:59.142774 5032 rpcz_store.cc:276] 0806 04:07:56.741803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.741997 (+ 194us) service_pool.cc:227] Handling call
0806 04:07:56.753961 (+ 11964us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:56.882454 (+128493us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:56.882484 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:56.882522 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:56.882563 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:58.983364 (+2100801us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400167954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:58.983410 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:58.983539 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":294930688}
I20260806 04:07:59.140405 5035 raft_consensus.cc:2362] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5. Re-sending same reply.
W20260806 04:07:59.235350 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1950) took 2379 ms (client timeout 1716 ms). Trace:
W20260806 04:07:59.235831 5035 rpcz_store.cc:269] 0806 04:07:56.856565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.856789 (+ 224us) service_pool.cc:227] Handling call
0806 04:07:58.982164 (+2125375us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:59.235257 (+253093us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6720256}
W20260806 04:07:59.152809 5030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1946) took 2330 ms (client timeout 2299 ms). Trace:
W20260806 04:07:59.236917 5030 rpcz_store.cc:269] 0806 04:07:56.811079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.811297 (+ 218us) service_pool.cc:227] Handling call
0806 04:07:58.982899 (+2171602us) spinlock_profiling.cc:243] Waited 64.2 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fab7c 00007f23d69fa73b 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:07:59.141616 (+158717us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":147682176}
I20260806 04:07:59.082581 7994 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:59.157675 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.149817 8084 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:07:59.246577 4720 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0f80 after lost signal to thread 8208
I20260806 04:07:59.249577 4817 raft_consensus.cc:1226] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:59.250873 7714 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:59.150352 8292 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.165447 5026 rpcz_store.cc:276] 0806 04:07:56.798930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.799050 (+ 120us) service_pool.cc:227] Handling call
0806 04:07:56.803311 (+ 4261us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:56.809458 (+ 6147us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:56.809487 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:56.818761 (+ 9274us) log.cc:842] Serialized 26 byte log entry
0806 04:07:57.041408 (+222647us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:57.041457 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:57.041550 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:57.147655 (+106105us) raft_consensus.cc:1671] finished
0806 04:07:57.147676 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:58.999969 (+1852293us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b540037d050. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:07:59.000228 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10252,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":5028,"spinlock_wait_cycles":8984320,"thread_start_us":203750,"threads_started":1,"wal-append.queue_time_us":308045,"wal-append.run_cpu_time_us":2452,"wal-append.run_wall_time_us":1003697}
I20260806 04:07:59.169051 5037 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 1.
I20260806 04:07:59.198793 7643 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.175892 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:59.175740 8299 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.177726 8306 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.150041 8302 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.155687 7721 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.156225 8212 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.048039 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.256827 8253 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:59.139895 8267 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:07:59.195148 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.202229 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1948) took 2374 ms. Trace:
I20260806 04:07:59.285841 4924 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:59.286013 5038 rpcz_store.cc:276] 0806 04:07:56.827872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.828009 (+ 137us) service_pool.cc:227] Handling call
0806 04:07:57.971632 (+1143623us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fab7c 00007f23d69fa73b 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:57.971848 (+ 216us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.201935 (+1230087us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.202154 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3845760}
W20260806 04:07:59.203898 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 117573us
I20260806 04:07:59.247213 8142 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:59.287492 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.213773 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.068594 8197 consensus_queue.cc:1046] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260806 04:07:59.239742 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.289880 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 336) took 2312 ms. Trace:
I20260806 04:07:59.234030 4669 catalog_manager.cc:5495] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:59.166319 4818 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:07:59.175298 4814 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:07:59.258596 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.209841 8069 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:07:59.210175 8304 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:07:59.271848 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.314250 4805 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:59.291603 4669 catalog_manager.cc:5495] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 3, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:59.272353 4925 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:07:59.198810 8180 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:07:59.271085 4882 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.319931 4996 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.321539 4996 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.270566 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:59.276934 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.228801 8090 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:07:59.281538 5030 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:07:59.282672 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:07:59.324368 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 103370us
I20260806 04:07:59.256182 4919 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:59.290442 4927 rpcz_store.cc:276] 0806 04:07:56.977065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.977219 (+ 154us) service_pool.cc:227] Handling call
0806 04:07:56.977624 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:57.038787 (+ 61163us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:57.038831 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:57.039272 (+ 441us) log.cc:842] Serialized 26 byte log entry
0806 04:07:57.190619 (+151347us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:07:57.190683 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:57.190741 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:07:57.918897 (+728156us) spinlock_profiling.cc:243] Waited 9.34 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:07:59.289464 (+1370567us) raft_consensus.cc:1671] finished
0806 04:07:59.289496 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:59.289806 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60674,"raft.queue_time_us":61529,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":8893,"spinlock_wait_cycles":21482240,"thread_start_us":86715,"threads_started":1,"wal-append.queue_time_us":228975}
W20260806 04:07:59.298950 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 85288.6us
W20260806 04:07:59.327061 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 101201us
I20260806 04:07:59.316370 4669 catalog_manager.cc:5495] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 2 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:59.323453 4813 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:59.327133 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.218113 8261 consensus_queue.cc:1046] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260806 04:07:59.329370 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.261324 8002 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260806 04:07:59.328402 4669 catalog_manager.cc:5495] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 3, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 04:07:59.331470 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 105709us
W20260806 04:07:59.334352 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 63361.1us
I20260806 04:07:59.335889 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.337543 5816 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:59.336596 4810 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 04:07:59.334673 4876 connection.cc:232] RPC call timeout handler was delayed by -2.0347912029828876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260806 04:07:59.332974 8303 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:59.334772 7720 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.284696 8172 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:07:59.345008 4876 connection.cc:232] RPC call timeout handler was delayed by -1.8484584731049836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.315s for a response
I20260806 04:07:59.260408 8084 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:59.350430 4876 connection.cc:232] RPC call timeout handler was delayed by -1.8245184631086886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
I20260806 04:07:59.345492 4669 catalog_manager.cc:5495] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 4 to 5, leader changed from d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 5 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:59.319237 8283 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:59.277801 7643 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.351500 4882 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:59.354599 4876 connection.cc:232] RPC call timeout handler was delayed by -1.7279951109085232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260806 04:07:59.355235 4876 connection.cc:232] RPC call timeout handler was delayed by -1.3319951109588146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
W20260806 04:07:59.355554 4876 connection.cc:232] RPC call timeout handler was delayed by -1.3017045420128852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
W20260806 04:07:59.355841 4876 connection.cc:232] RPC call timeout handler was delayed by -1.2864045421592891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
W20260806 04:07:59.356129 4876 connection.cc:232] RPC call timeout handler was delayed by -1.0559380771592259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
I20260806 04:07:59.352425 4669 catalog_manager.cc:5495] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:07:59.357380 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.211938 5854 rpcz_store.cc:276] 0806 04:07:56.803935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:56.809203 (+ 5268us) service_pool.cc:227] Handling call
0806 04:07:59.017272 (+2208069us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fab7c 00007f23d69fa73b 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:07:59.017504 (+ 232us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.088439 (+ 70935us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.090273 (+ 1834us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b54005604d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:59.090470 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174924928}
I20260806 04:07:59.325992 8090 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:59.368440 8090 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:59.312991 8142 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.353725 7720 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.372989 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.373546 7720 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.374471 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.375371 4930 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:07:59.376250 8084 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.376587 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.377398 4923 raft_consensus.cc:2379] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:07:59.377859 8084 consensus_queue.cc:1046] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:07:59.338248 8281 raft_consensus.cc:2741] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260806 04:07:59.333544 8068 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.379060 8068 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.380293 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.381101 8068 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.362998 8266 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.365124 4882 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.365736 8239 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:07:59.359175 8094 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.353619 8283 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:59.294777 8267 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.381767 5026 raft_consensus.cc:1226] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:59.382126 4927 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.362537 4924 raft_consensus.cc:1261] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260806 04:07:59.362929 8282 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.382467 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.450232 5038 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.382843 4931 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.362236 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.372628 8142 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:07:59.459364 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 105339us
I20260806 04:07:59.459688 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:59.392216 4768 connection.cc:232] RPC call timeout handler was delayed by -2.7873904418665916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
I20260806 04:07:59.334965 7932 raft_consensus.cc:3032] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:07:59.333520 8296 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:07:59.335139 8197 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:07:59.383227 4805 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:07:59.396225 4813 raft_consensus.cc:2396] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:07:59.397522 4926 raft_consensus.cc:1226] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:59.345461 8276 raft_consensus.cc:2741] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:07:59.334903 8261 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:07:59.401185 4920 raft_consensus.cc:1261] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:07:59.291816 7721 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.403762 5037 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260806 04:07:59.407076 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 76344.1us
I20260806 04:07:59.415225 8253 raft_consensus.cc:686] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:07:59.446885 8239 raft_consensus.cc:686] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:59.490554 8253 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.491396 8239 consensus_queue.cc:237] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:07:59.419652 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54602.5us
I20260806 04:07:59.495087 5032 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:59.415814 8330 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260806 04:07:59.419866 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.368755 8302 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.498956 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.499663 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.330137 8180 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.425191 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:07:59.425546 8319 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260806 04:07:59.430294 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.425220 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.502044 5029 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.367398 8295 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.363862 8212 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.367523 8306 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.444442 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.506255 5028 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.360647 8108 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:07:59.447047 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.359936 8303 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.093s
I20260806 04:07:59.462301 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.462632 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:07:59.463115 4768 connection.cc:232] RPC call timeout handler was delayed by -2.7441904419101775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
W20260806 04:07:59.510592 4768 connection.cc:232] RPC call timeout handler was delayed by -2.6271904418244958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
W20260806 04:07:59.511073 4768 connection.cc:232] RPC call timeout handler was delayed by -2.4858904418069869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20260806 04:07:59.511440 4768 connection.cc:232] RPC call timeout handler was delayed by -2.3283904418349266s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260806 04:07:59.511826 4768 connection.cc:232] RPC call timeout handler was delayed by -2.3040904419030994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
W20260806 04:07:59.512162 4768 connection.cc:232] RPC call timeout handler was delayed by -2.2221904417965561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
W20260806 04:07:59.512506 4768 connection.cc:232] RPC call timeout handler was delayed by -2.2077904418110847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22080000000000002s for a response
W20260806 04:07:59.512831 4768 connection.cc:232] RPC call timeout handler was delayed by -1.8045904419850558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2656s for a response
W20260806 04:07:59.513163 4768 connection.cc:232] RPC call timeout handler was delayed by -1.197090441826731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3331s for a response
I20260806 04:07:59.463465 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.357791 4669 catalog_manager.cc:5495] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 3 to 5, leader changed from d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 5 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:07:59.469125 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.444602 8283 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.474632 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.478247 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.359870 8224 consensus_queue.cc:1046] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20260806 04:07:59.495788 4931 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.499788 4809 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:59.425115 8172 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:07:59.395556 8140 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.442804 8094 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.465467 7932 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:59.514371 5038 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:59.514535 5036 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:07:59.431052 8266 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.508215 4927 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:59.520298 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.519256 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.531811 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.530354 5028 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.536154 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.537034 5038 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:07:59.539291 5029 raft_consensus.cc:1226] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260806 04:07:59.540849 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 77528.3us
I20260806 04:07:59.537520 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.538599 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.542870 4804 raft_consensus.cc:2379] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.543879 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.545591 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.545917 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 239) took 1981 ms. Trace:
I20260806 04:07:59.546340 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.547115 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:59.548635 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.549374 4814 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 3: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:59.550882 5036 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:07:59.470404 8112 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.555274 4930 raft_consensus.cc:2379] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.555800 5863 raft_consensus.cc:1226] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:59.553718 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.546299 5034 rpcz_store.cc:276] 0806 04:07:57.564173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.564363 (+ 190us) service_pool.cc:227] Handling call
0806 04:07:57.564708 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.037201 (+1472493us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:07:59.143971 (+106770us) log.cc:842] Serialized 14 byte log entry
0806 04:07:59.278523 (+134552us) log.cc:842] Serialized 14 byte log entry
0806 04:07:59.278759 (+ 236us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:59.278819 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:07:59.278883 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.545694 (+266811us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400321ed4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:07:59.545738 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:07:59.545862 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32559,"prepare.queue_time_us":99467,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":11453,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":4926,"spinlock_wait_cycles":3415582208,"thread_start_us":220573,"threads_started":2,"wal-append.queue_time_us":130340}
I20260806 04:07:59.560541 4808 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 04:07:59.559585 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.562362 4808 pending_rounds.cc:76] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 5
I20260806 04:07:59.556809 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.526156 8172 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:59.562721 4808 pending_rounds.cc:97] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.6
I20260806 04:07:59.574299 4808 raft_consensus.cc:2894] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Aborting config change with OpId 3.6: Aborted: Op aborted by new leader
I20260806 04:07:59.510222 8108 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:07:59.566686 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:07:59.566244 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.575591 5029 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:59.576040 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.557994 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.572286 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.570590 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:07:59.579126 4808 raft_consensus.cc:2866] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:07:59.579834 4809 raft_consensus.cc:2379] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 8: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:07:59.494943 8197 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.583659 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.585289 5034 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.586283 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.586474 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.586958 4926 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.587816 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.509711 8304 consensus_queue.cc:1035] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260806 04:07:59.589519 4925 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:59.589756 5812 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:07:59.590677 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:07:59.590958 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 61979.5us
I20260806 04:07:59.592594 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.593382 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.600312 4926 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.572707 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:07:59.595844 4810 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:07:59.595741 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.603148 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.603785 5038 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:59.604790 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.604266 5038 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
I20260806 04:07:59.606820 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.607105 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.607609 5028 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:07:59.607906 5036 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:59.608933 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.605448 5035 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:07:59.579270 8108 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:07:59.581597 8172 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:07:59.597568 8333 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260806 04:07:59.579324 8297 raft_consensus.cc:2741] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:07:59.613895 4926 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:07:59.614415 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.614996 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.616425 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.618971 5028 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:59.619058 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.619251 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.620275 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.620978 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.626106 4810 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.582031 8299 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.626523 5036 raft_consensus.cc:3037] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:59.628517 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.556082 8339 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260806 04:07:59.632503 4919 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:07:59.633034 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.633690 4814 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:07:59.604887 8281 consensus_queue.cc:1046] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
W20260806 04:07:59.633205 4919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 342) took 2186 ms (client timeout 1684 ms). Trace:
I20260806 04:07:59.635214 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:07:59.635673 4919 rpcz_store.cc:269] 0806 04:07:57.446296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:57.449237 (+ 2941us) service_pool.cc:227] Handling call
0806 04:07:59.633099 (+2183862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:07:59.636142 5028 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.636866 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.637368 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.638140 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.635222 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.638185 5028 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:59.639727 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.640239 5028 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:59.560613 8276 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 04:07:59.598531 8282 raft_consensus.cc:2741] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260806 04:07:59.643517 4927 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:07:59.649308 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:07:59.654330 4876 leader_election.cc:334] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.686s (SENT)
I20260806 04:07:59.667325 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:07:59.657335 5029 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:07:59.595217 8158 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:07:59.668984 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.670154 8341 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:07:59.670668 8341 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:07:59.671222 8341 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.671955 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.669947 5029 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:59.673256 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.673622 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.676884 4996 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.604498 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:07:59.688139 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.692377 5038 raft_consensus.cc:2362] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
I20260806 04:07:59.683220 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.693744 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.699704 4996 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.700774 5034 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.694713 5038 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:07:59.683312 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33546 (request call id 694) took 9844 ms. Trace:
I20260806 04:07:59.683490 5036 raft_consensus.cc:2454] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:07:59.716506 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.699918 5028 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:59.716698 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.716555 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.718046 4927 raft_consensus.cc:2396] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:07:59.598363 8090 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.719053 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:07:59.718257 5034 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:07:59.739709 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.601732 8295 consensus_queue.cc:1035] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260806 04:07:59.740185 5028 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:07:59.740979 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.661055 8180 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.746135 5028 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:59.707825 5853 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:07:59.623682 8084 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.085s
W20260806 04:07:59.744288 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 74820.1us
I20260806 04:07:59.748673 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.709115 4808 rpcz_store.cc:276] 0806 04:07:49.823048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:50.869010 (+1045962us) service_pool.cc:227] Handling call
0806 04:07:50.961693 (+ 92683us) log.cc:842] Serialized 494 byte log entry
0806 04:07:52.297421 (+1335728us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b54000963a0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d98f3416 00007f23d98ecdaa 00007f23d98ec014 00007f23d98d4545 00007f23d98d9771 00007f23d98d96f9 00007f23d98d9689 00007f23d98d9651 00007f23d98d881d 00007f23dfc96204
0806 04:07:59.646541 (+7349120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89887,"raft.queue_time_us":48790,"raft.run_cpu_time_us":10483,"raft.run_wall_time_us":1221054,"spinlock_wait_cycles":94374784,"thread_start_us":387929,"threads_started":2,"wal-append.queue_time_us":698870,"wal-append.run_cpu_time_us":3598,"wal-append.run_wall_time_us":1106056}
I20260806 04:07:59.688788 8339 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:07:59.685395 8299 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:07:59.757229 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.741828 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.758991 4809 raft_consensus.cc:1226] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:07:59.807297 5028 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:07:59.708240 8303 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:07:59.764480 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 167180us
I20260806 04:07:59.749918 5038 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
W20260806 04:07:59.830230 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 175970us
I20260806 04:07:59.834689 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:07:59.759672 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:07:59.609802 7932 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.682232 8158 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:07:59.838112 4992 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.986s (SENT) [suppressed 27 similar messages]
W20260806 04:07:59.834167 4876 leader_election.cc:334] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.716s (SENT)
I20260806 04:07:59.838929 4876 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.717108 8296 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.096s
I20260806 04:07:59.840471 5038 raft_consensus.cc:1226] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 04:07:59.812163 8094 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.842512 8164 consensus_queue.cc:1035] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:07:59.849187 5038 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:07:59.849783 4992 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.843755 8090 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.855212 8197 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.865079 5029 raft_consensus.cc:2396] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:07:59.855049 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:07:59.834265 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 244577us
I20260806 04:07:59.859495 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.868045 4810 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:07:59.868275 4817 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:07:59.869599 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:07:59.811893 8224 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:59.838246 5028 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:07:59.873731 8261 raft_consensus.cc:686] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:07:59.829901 7721 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.844365 8140 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:07:59.840369 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:07:59.894104 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:07:59.881361 8242 consensus_meta.cc:287] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.809s user 0.014s sys 0.000s
I20260806 04:07:59.875415 8261 consensus_queue.cc:237] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:07:59.949150 5038 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260806 04:07:59.952935 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 106731us
W20260806 04:07:59.952052 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 44487.4us
W20260806 04:07:59.956364 5030 consensus_meta.cc:287] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.504s user 0.016s sys 0.005s
W20260806 04:07:59.956066 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 45611.4us
W20260806 04:07:59.955708 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 61748.6us
I20260806 04:07:59.961329 5030 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:07:59.957515 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.879016 8112 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:07:59.959584 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:07:59.895833 8197 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:07:59.892155 8266 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:07:59.862323 8303 consensus_queue.cc:1035] T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
W20260806 04:07:59.843514 4657 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 086f8af9433f44aa8aa4fccbfa90eb02 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted by new leader
I20260806 04:07:59.905797 8192 consensus_queue.cc:1046] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260806 04:07:59.951416 8341 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:07:59.962437 8094 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:08:00.275401 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 265356us
I20260806 04:07:59.993338 4808 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:08:00.033573 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 74052.9us
W20260806 04:08:00.365814 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 531598us
I20260806 04:07:59.968676 7762 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:00.276139 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:08:00.032838 5038 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 04:08:00.478953 5034 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.002s sys 0.009s
W20260806 04:08:00.486613 4876 leader_election.cc:334] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.756s (SENT)
W20260806 04:08:00.366200 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 516476us
W20260806 04:08:00.211920 8365 log.cc:924] Time spent T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.179s user 0.000s sys 0.000s
W20260806 04:08:00.464177 8281 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.003s sys 0.008s
I20260806 04:08:00.532270 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:00.047112 7994 consensus_meta.cc:287] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.790s user 0.010s sys 0.002s
W20260806 04:08:00.529518 5034 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.004s sys 0.010s
W20260806 04:08:00.211799 8353 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.256s user 0.000s sys 0.000s
W20260806 04:08:00.214138 8087 consensus_meta.cc:287] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.058s user 0.010s sys 0.000s
W20260806 04:08:00.513751 4804 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.012s sys 0.042s
I20260806 04:08:00.046965 8090 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
W20260806 04:08:00.213747 8352 log.cc:924] Time spent T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.257s user 0.000s sys 0.000s
W20260806 04:07:59.987181 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 121122us
I20260806 04:07:59.993059 8140 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:08:00.530537 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:00.259558 8336 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
W20260806 04:08:00.003674 4657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 160452us
W20260806 04:08:00.540520 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 53880.1us
I20260806 04:07:59.970427 8068 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:00.544064 4804 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.013s sys 0.043s
I20260806 04:08:00.548273 5034 raft_consensus.cc:1261] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:08:00.549973 5034 raft_consensus.cc:1226] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 04:08:00.552013 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.960150 8242 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:00.546236 4876 leader_election.cc:334] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.807s (SENT)
W20260806 04:08:00.562817 5026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.045s user 0.032s sys 0.052s
W20260806 04:08:00.563513 5026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.045s user 0.032s sys 0.052s
I20260806 04:08:00.565819 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:00.541508 8281 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.593s user 0.004s sys 0.010s
I20260806 04:08:00.487213 8094 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:00.574348 4876 leader_election.cc:334] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.196s (SENT)
I20260806 04:08:00.481501 8158 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:00.575412 4876 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.574334 4923 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:00.573307 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:00.530773 8355 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:00.580346 5034 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:08:00.585098 4808 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:00.588286 4928 raft_consensus.cc:2379] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:00.601442 5853 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.789s user 0.007s sys 0.003s
I20260806 04:08:00.602106 5853 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:00.603394 5866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 569) took 1252 ms. Trace:
I20260806 04:08:00.603550 8197 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:00.604074 8328 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.000s sys 0.010s
W20260806 04:08:00.604527 8328 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s user 0.000s sys 0.010s
W20260806 04:08:00.605054 8344 log.cc:924] Time spent T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.570s user 0.001s sys 0.000s
I20260806 04:08:00.604557 8197 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:00.605759 8320 log.cc:924] Time spent T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
W20260806 04:08:00.607054 8364 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260806 04:08:00.604100 5866 rpcz_store.cc:276] 0806 04:07:59.351095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.351239 (+ 144us) service_pool.cc:227] Handling call
0806 04:07:59.367025 (+ 15786us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.373054 (+ 6029us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:07:59.373089 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:59.373129 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:59.373161 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:00.602444 (+1229283us) spinlock_profiling.cc:243] Waited 5.79 ms on lock 0x7b54007e04d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:00.602530 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.603288 (+ 758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13314432}
I20260806 04:08:00.562223 7762 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:00.608973 8289 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:00.608237 8334 raft_consensus.cc:3032] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:08:00.618539 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:00.257936 7721 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.314s
I20260806 04:08:00.616185 8389 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:00.633435 5812 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:07:59.972122 8354 consensus_queue.cc:1046] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.609s
I20260806 04:08:00.130035 8369 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.459s
I20260806 04:08:00.578383 8094 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:00.623929 8158 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:00.637120 8358 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.001s sys 0.037s
I20260806 04:08:00.637504 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:00.587436 7994 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:00.654246 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1661) took 1174 ms. Trace:
I20260806 04:08:00.675757 5037 rpcz_store.cc:276] 0806 04:07:59.480226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.503873 (+ 23647us) service_pool.cc:227] Handling call
0806 04:07:59.504292 (+ 419us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:59.540145 (+ 35853us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.540178 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:59.551761 (+ 11583us) log.cc:842] Serialized 26 byte log entry
0806 04:07:59.744514 (+192753us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:07:59.744716 (+ 202us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.744812 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:00.653972 (+909160us) raft_consensus.cc:1671] finished
0806 04:08:00.653998 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.654171 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40710,"raft.queue_time_us":13881,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":16227,"thread_start_us":177775,"threads_started":1,"wal-append.queue_time_us":271898}
I20260806 04:08:00.680341 8242 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:00.659196 8358 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.617s user 0.001s sys 0.038s
I20260806 04:08:00.653411 5034 raft_consensus.cc:2396] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 04:08:00.677747 8283 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.136s user 0.011s sys 0.000s
W20260806 04:08:00.629443 5853 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 294) took 1291 ms (client timeout 1653 ms). Trace:
I20260806 04:08:00.674072 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1613) took 1253 ms. Trace:
I20260806 04:08:00.679106 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 358) took 1342 ms. Trace:
I20260806 04:08:00.702420 4818 rpcz_store.cc:276] 0806 04:07:59.336462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.339071 (+ 2609us) service_pool.cc:227] Handling call
0806 04:07:59.360212 (+ 21141us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:59.398579 (+ 38367us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.398623 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:59.428230 (+ 29607us) log.cc:842] Serialized 26 byte log entry
0806 04:07:59.506615 (+ 78385us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:07:59.506685 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:07:59.506753 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:00.678317 (+1171564us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b54000927a0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d98e8771 00007f23d98f3220 00007f23d98f31c8 00007f23d98f3148 00007f23d98f3103 00007f23d98f238c 00007f23df963206 00007f23df95b808 00007f23d992f38b 00007f23d9931e80
0806 04:08:00.678670 (+ 353us) raft_consensus.cc:1671] finished
0806 04:08:00.678701 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.679026 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"spinlock_wait_cycles":13367040,"thread_start_us":67536,"threads_started":1,"wal-append.queue_time_us":70857}
W20260806 04:08:00.690582 4810 consensus_meta.cc:287] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.822s user 0.010s sys 0.001s
I20260806 04:08:00.705374 4810 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:00.672856 5860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 636) took 1519 ms. Trace:
W20260806 04:08:00.675129 8172 consensus_meta.cc:287] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.015s user 0.011s sys 0.000s
I20260806 04:08:00.629143 8069 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:00.630273 8334 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:00.632282 8112 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:00.706504 5860 rpcz_store.cc:276] 0806 04:07:59.153551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.153685 (+ 134us) service_pool.cc:227] Handling call
0806 04:07:59.157670 (+ 3985us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.471209 (+313539us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.471246 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:07:59.471302 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:07:59.471377 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:00.672534 (+1201157us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b540051ed54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:00.672593 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.672784 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":720411904}
I20260806 04:08:00.702318 4926 rpcz_store.cc:276] 0806 04:07:59.420380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.604019 (+183639us) service_pool.cc:227] Handling call
0806 04:07:59.604340 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.623198 (+ 18858us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:00.673795 (+1050597us) spinlock_profiling.cc:243] Waited 7.04 ms on lock 0x7b5400413cd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:00.674008 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16199040}
W20260806 04:08:00.712520 4992 leader_election.cc:334] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.574s (SENT)
I20260806 04:08:00.737744 4992 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.651530 8372 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:00.701663 5853 rpcz_store.cc:269] 0806 04:07:59.332507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.559249 (+226742us) service_pool.cc:227] Handling call
0806 04:08:00.624032 (+1064783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:00.634274 7762 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:00.726110 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1658) took 1277 ms. Trace:
I20260806 04:08:00.724087 8341 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:00.617029 8355 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:00.771944 5026 rpcz_store.cc:276] 0806 04:07:59.448552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.448748 (+ 196us) service_pool.cc:227] Handling call
0806 04:07:59.449198 (+ 450us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:59.465392 (+ 16194us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.465438 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:59.475424 (+ 9986us) log.cc:842] Serialized 26 byte log entry
0806 04:08:00.563972 (+1088548us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:08:00.564010 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:00.564083 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:00.725842 (+161759us) raft_consensus.cc:1671] finished
0806 04:08:00.725868 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.726038 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8258,"raft.queue_time_us":12763,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":28660,"thread_start_us":1045824,"threads_started":1,"wal-append.queue_time_us":1240629}
I20260806 04:08:00.711014 8158 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:00.675705 8197 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:00.784086 5030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.004s sys 0.017s
I20260806 04:08:00.738875 8330 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:00.635033 8289 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:00.787729 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 154385us
W20260806 04:08:00.833163 5030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.004s sys 0.017s
I20260806 04:08:00.708425 8283 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:00.697652 7994 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:00.710939 8172 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:00.876655 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 279208us
W20260806 04:08:00.877460 4876 leader_election.cc:334] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.210s (SENT)
I20260806 04:08:00.738256 8069 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:00.881071 4876 leader_election.cc:334] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.211s (SENT)
I20260806 04:08:00.881232 4813 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:00.881696 4876 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.886644 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:00.887210 8374 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.303s user 0.000s sys 0.000s
I20260806 04:08:00.887310 8283 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:00.888161 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:00.889714 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 360) took 1214 ms. Trace:
I20260806 04:08:00.891163 8371 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:00.891031 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:00.891739 8371 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:00.892787 4876 leader_election.cc:334] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.227s (SENT)
I20260806 04:08:00.893224 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.893409 8197 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:00.893869 8371 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:00.894136 8197 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:00.898761 4876 leader_election.cc:334] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 4.605s (SENT)
W20260806 04:08:00.899012 8299 consensus_meta.cc:287] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.065s user 0.011s sys 0.001s
I20260806 04:08:00.899474 8299 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:00.900570 4876 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.890062 4804 rpcz_store.cc:276] 0806 04:07:59.675588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.675859 (+ 271us) service_pool.cc:227] Handling call
0806 04:07:59.676257 (+ 398us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:59.740490 (+ 64233us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.740524 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:59.753196 (+ 12672us) log.cc:842] Serialized 26 byte log entry
0806 04:08:00.546954 (+793758us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:00.546996 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:00.547089 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:00.889512 (+342423us) raft_consensus.cc:1671] finished
0806 04:08:00.889533 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.889658 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54850,"raft.queue_time_us":169716,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":35222,"thread_start_us":757826,"threads_started":2,"wal-append.queue_time_us":907175}
I20260806 04:08:00.843159 8330 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:00.906554 4931 raft_consensus.cc:2354] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:00.894382 8371 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:00.914598 4992 leader_election.cc:334] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.696s (SENT)
I20260806 04:08:00.930141 4992 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.792210 8389 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:00.886739 8390 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260806 04:08:00.830967 8158 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:00.851408 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:00.951979 4879 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.958674 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:00.959570 5026 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:00.961983 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:00.891053 8403 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:00.782975 8372 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:00.928956 8330 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:00.964890 4879 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.878980 8378 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.058s
I20260806 04:08:00.972705 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.930104 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:00.977140 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1988) took 1377 ms. Trace:
I20260806 04:08:00.985942 5030 rpcz_store.cc:276] 0806 04:07:59.600777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.972428 (+371651us) service_pool.cc:227] Handling call
0806 04:07:59.972894 (+ 466us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:59.973193 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.973231 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:59.973812 (+ 581us) log.cc:842] Serialized 26 byte log entry
0806 04:08:00.844919 (+871107us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:00.973235 (+128316us) log.cc:842] Serialized 14 byte log entry
0806 04:08:00.976470 (+ 3235us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:00.976694 (+ 224us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:00.976808 (+ 114us) raft_consensus.cc:1671] finished
0806 04:08:00.976819 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:00.977066 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4395,"thread_start_us":890362,"threads_started":2,"wal-append.queue_time_us":861405}
W20260806 04:08:00.990722 4992 leader_election.cc:334] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.976s (SENT)
I20260806 04:08:00.991340 4992 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:00.996919 5026 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:00.998593 5812 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:00.932682 8197 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260806 04:08:00.935199 8112 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260806 04:08:00.935611 8371 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:00.841938 8334 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:00.974201 4919 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:00.983847 8239 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:01.000010 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:01.012974 8108 consensus_meta.cc:287] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.348s user 0.014s sys 0.000s
I20260806 04:08:01.018359 5853 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:01.016629 8403 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:00.940366 8389 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:01.000356 8261 consensus_queue.cc:1046] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260806 04:08:01.029162 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 64928.9us
I20260806 04:08:01.028342 8239 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:01.162783 8401 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:08:01.164182 8239 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:01.120749 8345 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:01.113636 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:01.123813 8374 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.094s user 0.001s sys 0.005s
I20260806 04:08:01.223075 4876 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:01.077211 8200 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:01.026141 8108 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:01.170869 8405 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:01.121747 8120 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:01.152206 8403 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.167171 8355 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:01.738406 8345 consensus_meta.cc:287] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.005s
I20260806 04:08:01.680835 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 250) took 1122 ms. Trace:
W20260806 04:08:01.356740 4931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1616) took 1910 ms (client timeout 1909 ms). Trace:
I20260806 04:08:01.172335 8395 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:01.354352 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 361) took 1678 ms. Trace:
I20260806 04:08:01.288908 8158 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:01.819675 8395 raft_consensus.cc:686] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:01.384189 8295 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:01.329918 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1483) took 1971 ms. Trace:
W20260806 04:08:01.224462 8094 consensus_meta.cc:287] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.526s user 0.011s sys 0.000s
I20260806 04:08:01.229228 8200 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:01.354539 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2155) took 1457 ms. Trace:
I20260806 04:08:01.174866 7994 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:01.319144 8415 log.cc:924] Time spent T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260806 04:08:01.169430 8371 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:01.813004 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 1.28087e+06us
I20260806 04:08:01.849854 5866 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
W20260806 04:08:01.172156 8339 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.338s user 0.006s sys 0.006s
W20260806 04:08:01.496433 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 140037us
W20260806 04:08:01.531291 4992 leader_election.cc:334] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 2.182s (SENT)
I20260806 04:08:01.740653 5037 rpcz_store.cc:276] 0806 04:08:00.558112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.682958 (+124846us) service_pool.cc:227] Handling call
0806 04:08:00.683361 (+ 403us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:00.831783 (+148422us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:00.831825 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:00.831879 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:00.831920 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.680516 (+848596us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540008e654. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:01.680575 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.680751 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333089024}
I20260806 04:08:01.239400 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:01.243489 8069 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:01.821699 4925 rpcz_store.cc:276] 0806 04:07:59.358364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.358501 (+ 137us) service_pool.cc:227] Handling call
0806 04:07:59.360454 (+ 1953us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.598966 (+238512us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.329475 (+1730509us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54001b76d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:01.329766 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":515320192}
I20260806 04:08:01.830531 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 355) took 1064 ms. Trace:
W20260806 04:08:01.821302 4931 rpcz_store.cc:269] 0806 04:07:59.446533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.531535 (+ 85002us) service_pool.cc:227] Handling call
0806 04:08:01.356536 (+1825001us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54004137d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:01.356663 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3156567424}
W20260806 04:08:01.823678 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 858824us
I20260806 04:08:01.832409 4817 rpcz_store.cc:276] 0806 04:07:59.897242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.897534 (+ 292us) service_pool.cc:227] Handling call
0806 04:07:59.897972 (+ 438us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.898261 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:00.061760 (+163499us) log.cc:842] Serialized 14 byte log entry
0806 04:08:01.354098 (+1292338us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:01.354168 (+ 70us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:01.354229 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.354319 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.354471 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1298786,"prepare.queue_time_us":313719,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":1293473,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":109266,"threads_started":1}
I20260806 04:08:01.453985 8283 raft_consensus.cc:3032] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:01.738072 8362 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.874853 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:01.874424 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 359) took 1085 ms. Trace:
I20260806 04:08:01.314317 8239 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:01.737494 8348 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.740449 8345 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:01.888154 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:01.354898 7419 raft_consensus.cc:3032] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 04:08:01.859339 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 328094us
I20260806 04:08:01.349646 8372 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:01.174321 8417 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:01.893700 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:01.893528 8417 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:01.894719 4925 raft_consensus.cc:2396] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 04:08:01.894917 4930 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1611) took 2510 ms (client timeout 1633 ms). Trace:
I20260806 04:08:01.895237 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 358) took 1107 ms. Trace:
I20260806 04:08:01.895617 4925 rpcz_store.cc:276] 0806 04:08:00.788483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.873637 (+1085154us) service_pool.cc:227] Handling call
0806 04:08:01.895178 (+ 21541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:01.895953 8417 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:01.391482 8087 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:01.896543 8087 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:01.897167 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 360) took 1107 ms. Trace:
I20260806 04:08:01.897430 8087 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:01.898195 4920 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:08:01.318722 8292 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:01.243666 8413 log.cc:924] Time spent T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
I20260806 04:08:01.391101 8090 raft_consensus.cc:686] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:01.347821 8140 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:01.401300 8291 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:01.349472 7643 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:01.305071 8242 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:01.622718 8281 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:01.863000 4926 rpcz_store.cc:276] 0806 04:08:00.765834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.766033 (+ 199us) service_pool.cc:227] Handling call
0806 04:08:00.766423 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:00.882237 (+115814us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:00.882278 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:00.882325 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:00.882363 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.830292 (+947929us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54003543d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:01.830340 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.830471 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265832192}
W20260806 04:08:01.847333 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 624328us
I20260806 04:08:01.968921 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 361) took 1178 ms. Trace:
I20260806 04:08:01.969463 4926 rpcz_store.cc:276] 0806 04:08:00.791695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.967919 (+1176224us) service_pool.cc:227] Handling call
0806 04:08:01.968305 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.968497 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:01.968529 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:01.968567 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:01.968616 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.968680 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.968845 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:01.868372 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:01.819262 4814 rpcz_store.cc:276] 0806 04:07:59.676148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.676317 (+ 169us) service_pool.cc:227] Handling call
0806 04:07:59.676603 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.676807 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:07:59.895203 (+218396us) log.cc:842] Serialized 14 byte log entry
0806 04:08:00.042163 (+146960us) log.cc:842] Serialized 14 byte log entry
0806 04:08:01.353744 (+1311581us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b18003ed940. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23d99523d0 00007f23d9964576 00007f23d99750d7 00007f23d9975058
0806 04:08:01.353977 (+ 233us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:01.354038 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:01.354088 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.354168 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.354287 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":224218,"prepare.queue_time_us":485837,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":82,"raft.queue_time_us":1464460,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":290,"spinlock_wait_cycles":18477952,"thread_start_us":186453,"threads_started":1}
I20260806 04:08:01.871398 5035 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 04:08:01.971470 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 103215us
I20260806 04:08:01.971275 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 260) took 1014 ms. Trace:
I20260806 04:08:01.972550 5035 rpcz_store.cc:276] 0806 04:08:00.956469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.829139 (+872670us) service_pool.cc:227] Handling call
0806 04:08:01.864668 (+ 35529us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.971025 (+106357us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.971227 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:01.972815 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:01.974076 4768 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:01.622437 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:01.243924 8399 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:01.238142 7762 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:01.681744 8358 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:01.867865 7994 consensus_queue.cc:1046] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260806 04:08:01.910316 8345 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.329542 8444 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.875540 4817 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:01.875437 4931 rpcz_store.cc:276] 0806 04:08:00.789543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.873704 (+1084161us) service_pool.cc:227] Handling call
0806 04:08:01.873979 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.874135 (+ 156us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:01.874156 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:01.874200 (+ 44us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:01.874227 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.874277 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.874372 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:01.991374 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:01.992041 4817 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:01.993201 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:01.723831 8330 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.723s user 0.010s sys 0.003s
I20260806 04:08:01.874285 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 363) took 1033 ms. Trace:
I20260806 04:08:01.876582 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:01.242687 8408 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:01.887626 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:01.846098 8094 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:01.895323 4930 rpcz_store.cc:269] 0806 04:07:59.384454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.384679 (+ 225us) service_pool.cc:227] Handling call
0806 04:08:01.894644 (+2509965us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b54000f6754. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:01.894832 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123933568}
I20260806 04:08:01.889001 4808 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:01.852633 8405 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:01.885306 8362 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:01.900521 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2148) took 2253 ms. Trace:
I20260806 04:08:01.901216 7714 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.826431 8403 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:01.897569 4925 rpcz_store.cc:276] 0806 04:08:00.789980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.896330 (+1106350us) service_pool.cc:227] Handling call
0806 04:08:01.896650 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.896839 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:01.896862 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:01.896903 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:01.896957 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.897011 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.897115 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:01.863216 8449 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.866894 8354 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:01.866739 8192 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:01.866534 8339 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:08:01.911366 5038 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.013s sys 0.027s
W20260806 04:08:01.929349 4995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.027s sys 0.039s
I20260806 04:08:01.874399 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 298) took 2309 ms. Trace:
I20260806 04:08:01.868600 8283 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:01.382226 8412 consensus_queue.cc:1046] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.509s
I20260806 04:08:01.962795 8291 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:01.892145 7419 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:01.844233 8295 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:01.349155 8428 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.178s user 0.001s sys 0.004s
I20260806 04:08:01.342272 8401 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:01.986357 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:01.962080 8140 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:08:02.032734 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2145) took 2534 ms. Trace:
I20260806 04:08:02.059767 4813 rpcz_store.cc:276] 0806 04:07:59.498058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.498223 (+ 165us) service_pool.cc:227] Handling call
0806 04:07:59.498533 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:00.881829 (+1383296us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.032457 (+1150628us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54003c1c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:02.032666 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3176727424}
W20260806 04:08:02.048877 4995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.031s sys 0.043s
I20260806 04:08:02.048489 4809 rpcz_store.cc:276] 0806 04:07:59.647198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.647660 (+ 462us) service_pool.cc:227] Handling call
0806 04:07:59.648112 (+ 452us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.830670 (+182558us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.900266 (+2069596us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540039a3d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:01.900438 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254411520}
W20260806 04:08:02.049381 5038 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.014s sys 0.028s
W20260806 04:08:02.037762 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1656) took 2602 ms (client timeout 1590 ms). Trace:
I20260806 04:08:02.064960 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:02.065223 5032 rpcz_store.cc:269] 0806 04:07:59.435335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.435501 (+ 166us) service_pool.cc:227] Handling call
0806 04:08:02.037566 (+2602065us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b54003214d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:02.037683 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79723776}
I20260806 04:08:01.990233 8351 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:02.025831 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 57621.8us
I20260806 04:08:02.032032 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1980) took 2434 ms. Trace:
I20260806 04:08:02.076203 4876 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:02.056905 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:02.056869 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 153923us
I20260806 04:08:01.994887 7643 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:02.065642 4809 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:02.066437 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1332) took 2433 ms. Trace:
I20260806 04:08:02.051139 4919 rpcz_store.cc:276] 0806 04:08:00.840481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.239064 (+398583us) service_pool.cc:227] Handling call
0806 04:08:01.874228 (+635164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:02.080493 4876 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.052444 4810 raft_consensus.cc:2396] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260806 04:08:02.055140 8291 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:01.994889 8297 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:08:02.087213 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.006s sys 0.021s
I20260806 04:08:02.074311 4926 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:02.087603 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.505s user 0.006s sys 0.022s
I20260806 04:08:02.045078 8108 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.088831 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:02.075923 5036 rpcz_store.cc:276] 0806 04:07:59.597448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.832619 (+235171us) service_pool.cc:227] Handling call
0806 04:08:00.652647 (+820028us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:00.652876 (+ 229us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:00.653281 (+ 405us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:00.653310 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:00.653834 (+ 524us) log.cc:842] Serialized 26 byte log entry
0806 04:08:00.884042 (+230208us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:01.021660 (+137618us) log.cc:842] Serialized 14 byte log entry
0806 04:08:01.458212 (+436552us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.458331 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:02.031701 (+573370us) raft_consensus.cc:1671] finished
0806 04:08:02.031806 (+ 105us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.031967 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1922,"prepare.queue_time_us":237078,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":150435,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":10797,"spinlock_wait_cycles":3842304,"thread_start_us":711030,"threads_started":3,"wal-append.queue_time_us":324996}
I20260806 04:08:02.089480 5860 raft_consensus.cc:2362] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7. Re-sending same reply.
I20260806 04:08:02.073438 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.079596 4816 rpcz_store.cc:276] 0806 04:07:59.632990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.633556 (+ 566us) service_pool.cc:227] Handling call
0806 04:07:59.633881 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:07:59.674482 (+ 40601us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:07:59.674518 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:07:59.708207 (+ 33689us) log.cc:842] Serialized 26 byte log entry
0806 04:07:59.841702 (+133495us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:07:59.841760 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.057001 (+2215241us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b54000afad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:02.057055 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:02.057105 (+ 50us) raft_consensus.cc:1671] finished
0806 04:08:02.057117 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.066379 (+ 9262us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15395,"raft.queue_time_us":137984,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":16585,"spinlock_wait_cycles":64059264,"thread_start_us":93774,"threads_started":1,"wal-append.queue_time_us":160389,"wal-append.run_cpu_time_us":27272,"wal-append.run_wall_time_us":1782134}
I20260806 04:08:02.090694 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:02.091322 4927 raft_consensus.cc:2362] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6. Re-sending same reply.
I20260806 04:08:02.085450 4818 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260806 04:08:02.091784 4927 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 357) took 1325 ms (client timeout 1684 ms). Trace:
W20260806 04:08:02.092180 4927 rpcz_store.cc:269] 0806 04:08:00.787414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.831591 (+ 44177us) service_pool.cc:227] Handling call
0806 04:08:02.090503 (+1258912us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:02.091732 (+ 1229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5497088}
I20260806 04:08:01.952782 8292 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.059826 8296 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:02.095147 4814 raft_consensus.cc:2396] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:02.095515 4816 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: [3.3-4.4]
I20260806 04:08:02.096833 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:02.083568 8351 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:02.147893 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.063140 8192 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.155325 5812 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:02.112280 8283 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.111320 8090 consensus_queue.cc:237] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.058171 8449 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.155589 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:02.178869 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 76310.3us
I20260806 04:08:02.179157 4919 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:02.156536 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:02.073697 8069 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:02.159197 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.159128 4882 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:02.085940 8399 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:08:02.161836 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 65161.3us
I20260806 04:08:01.985742 8281 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:01.892441 8348 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:01.995996 8345 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.167084 8242 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:02.206241 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:02.206673 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 221032us
I20260806 04:08:02.045425 8453 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
W20260806 04:08:02.173717 4992 leader_election.cc:334] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.958s (SENT)
W20260806 04:08:02.175209 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 94816.6us
I20260806 04:08:02.115020 8239 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:02.216110 4992 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:02.177752 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:02.110558 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:02.192144 4816 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
I20260806 04:08:01.989967 8358 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.054880 8354 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.054018 8330 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.201786 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1990) took 2600 ms. Trace:
I20260806 04:08:02.203604 8459 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:02.164206 8351 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:02.049654 8444 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.205757 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:02.157414 7419 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:01.968312 8395 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.172489 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:02.215222 4768 leader_election.cc:398] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:02.253682 4768 leader_election.cc:401] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:08:02.219722 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.192006 8108 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:08:02.256922 5813 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.050s sys 0.078s
I20260806 04:08:02.247052 8327 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:02.230557 5866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 305) took 1283 ms. Trace:
I20260806 04:08:02.190876 7994 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:02.201939 8069 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.259346 5866 rpcz_store.cc:276] 0806 04:08:00.947234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.947438 (+ 204us) service_pool.cc:227] Handling call
0806 04:08:00.980825 (+ 33387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.850395 (+869570us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.230230 (+379835us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b54005eae54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:02.230476 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1998015232}
I20260806 04:08:02.067600 8401 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.194715 8399 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:02.237720 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 55963us
I20260806 04:08:02.194087 8446 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:02.059163 7714 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.241866 5038 rpcz_store.cc:276] 0806 04:07:59.601363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.544914 (+943551us) service_pool.cc:227] Handling call
0806 04:08:00.575202 (+ 30288us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:00.742733 (+167531us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:00.742763 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:00.771356 (+ 28593us) log.cc:842] Serialized 26 byte log entry
0806 04:08:00.984722 (+213366us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:01.309205 (+324483us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.064940 (+755735us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.065016 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:02.201559 (+136543us) raft_consensus.cc:1671] finished
0806 04:08:02.201578 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.201722 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9704,"prepare.queue_time_us":182910,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":564790,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":52047,"thread_start_us":1305407,"threads_started":4,"wal-append.queue_time_us":328038}
I20260806 04:08:02.172668 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:02.270116 4814 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:02.160830 8296 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.276808 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:02.206066 8339 consensus_queue.cc:1046] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 04:08:02.193503 8456 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.079s
W20260806 04:08:02.257436 8438 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260806 04:08:02.279316 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.250778 8094 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.193992 8283 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:02.265053 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:02.269053 4876 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.268836 8399 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:02.261988 8459 raft_consensus.cc:686] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:02.295001 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:02.273310 8345 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:02.273515 8330 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.273627 7419 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:02.275601 8401 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.268327 8289 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:02.227970 8239 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.282459 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:02.283735 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 191463us
I20260806 04:08:02.285952 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.273018 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:02.281013 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 365) took 1310 ms. Trace:
I20260806 04:08:02.317691 8459 consensus_queue.cc:237] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.328783 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:02.334502 4931 rpcz_store.cc:276] 0806 04:08:00.971056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.991044 (+1019988us) service_pool.cc:227] Handling call
0806 04:08:01.991347 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.991551 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:02.195349 (+203798us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.280553 (+ 85204us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:02.280628 (+ 75us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:02.280696 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.280793 (+ 97us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.280941 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13169,"prepare.queue_time_us":242855,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":12627,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":11863,"thread_start_us":236826,"threads_started":2}
I20260806 04:08:02.267949 8463 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:02.334774 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 48938.7us
I20260806 04:08:02.269421 8446 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:02.339139 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 165427us
W20260806 04:08:02.346416 4992 leader_election.cc:334] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 2.178s (SENT)
I20260806 04:08:02.347095 4992 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:02.348121 4805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 355) took 3014 ms (client timeout 2409 ms). Trace:
W20260806 04:08:02.348551 4805 rpcz_store.cc:269] 0806 04:07:59.334212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.335554 (+ 1342us) service_pool.cc:227] Handling call
0806 04:08:02.347882 (+3012328us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b5400230d54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:02.348030 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99647232}
I20260806 04:08:02.291129 7994 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:02.350827 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:02.355146 4931 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:02.358429 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 303) took 1473 ms. Trace:
I20260806 04:08:02.315379 8405 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:02.399304 4927 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:02.434430 4925 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:02.423372 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 257) took 1467 ms. Trace:
I20260806 04:08:02.381455 8242 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:02.421974 8486 raft_consensus.cc:3032] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:02.464768 8486 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:02.351855 8358 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:02.415180 7714 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.407147 4814 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:02.363503 5859 rpcz_store.cc:276] 0806 04:08:00.884860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.885044 (+ 184us) service_pool.cc:227] Handling call
0806 04:08:00.885353 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:00.885552 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:01.025129 (+139577us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.353885 (+1328756us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b180044ed80. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23dd32f167 00007f23d9964807 00007f23d9973f0f 00007f23d9973e88
0806 04:08:02.357739 (+ 3854us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.358001 (+ 262us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:02.358067 (+ 66us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:02.358127 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.358192 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.358356 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8699,"prepare.queue_time_us":1468427,"prepare.run_wall_time_us":42,"raft.queue_time_us":47660,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":13519,"spinlock_wait_cycles":151560064,"thread_start_us":169620,"threads_started":2,"wal-append.queue_time_us":294955,"wal-append.run_wall_time_us":1029002}
W20260806 04:08:02.360404 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 141771us
W20260806 04:08:02.469399 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 62303.9us
I20260806 04:08:02.415917 5862 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
W20260806 04:08:02.476375 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 138655us
I20260806 04:08:02.476959 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 645) took 2932 ms. Trace:
I20260806 04:08:02.458887 5030 rpcz_store.cc:276] 0806 04:08:00.955566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.987977 (+ 32411us) service_pool.cc:227] Handling call
0806 04:08:01.009829 (+ 21852us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:01.012526 (+ 2697us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:01.012552 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:01.016018 (+ 3466us) log.cc:842] Serialized 26 byte log entry
0806 04:08:01.222124 (+206106us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:01.222162 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.222254 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:02.423095 (+1200841us) raft_consensus.cc:1671] finished
0806 04:08:02.423112 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.423294 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2202,"raft.queue_time_us":65843,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":45579,"thread_start_us":196294,"threads_started":1,"wal-append.queue_time_us":1040699}
I20260806 04:08:02.053961 5857 rpcz_store.cc:276] 0806 04:07:59.559093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.565769 (+ 6676us) service_pool.cc:227] Handling call
0806 04:08:00.681611 (+1115842us) spinlock_profiling.cc:243] Waited 9.67 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:00.701334 (+ 19723us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:00.776005 (+ 74671us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:00.781695 (+ 5690us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:00.929914 (+148219us) log.cc:842] Serialized 26 byte log entry
0806 04:08:01.058900 (+128986us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:01.076861 (+ 17961us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:01.120431 (+ 43570us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:01.352212 (+231781us) raft_consensus.cc:1671] finished
0806 04:08:01.352286 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:01.868640 (+516354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26365,"raft.queue_time_us":52285,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":9749,"spinlock_wait_cycles":22230912,"thread_start_us":96319,"threads_started":1,"wal-append.queue_time_us":391435}
I20260806 04:08:02.480014 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:02.480836 5036 raft_consensus.cc:2379] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:02.479543 4882 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.477311 5862 rpcz_store.cc:276] 0806 04:07:59.544088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.544211 (+ 123us) service_pool.cc:227] Handling call
0806 04:08:02.415424 (+2871213us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:02.415690 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.476705 (+ 61015us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.476904 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5483904}
I20260806 04:08:02.422477 8460 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:02.355093 8444 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:02.524714 4768 connection.cc:232] RPC call timeout handler was delayed by -1.5232940011192113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
I20260806 04:08:02.526486 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:02.527244 5035 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:02.527083 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:02.528247 5038 raft_consensus.cc:2379] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
W20260806 04:08:02.462525 8482 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
W20260806 04:08:02.530592 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 51145.3us
I20260806 04:08:02.531530 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:02.347317 8446 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:02.534353 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:02.525511 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4782940011937171s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
W20260806 04:08:02.535143 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4728940012864769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20260806 04:08:02.535200 5032 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
W20260806 04:08:02.535487 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4656940011773258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
W20260806 04:08:02.535904 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4647940010763705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260806 04:08:02.361223 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:02.536219 4768 connection.cc:232] RPC call timeout handler was delayed by -1.3351940012071282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1734s for a response
I20260806 04:08:02.536417 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:02.343222 8330 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.151s
I20260806 04:08:02.540625 5030 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:08:02.541149 4768 connection.cc:232] RPC call timeout handler was delayed by -1.2703940011560917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
W20260806 04:08:02.552785 4768 connection.cc:232] RPC call timeout handler was delayed by -1.2640940011478961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
W20260806 04:08:02.543157 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 251650us
W20260806 04:08:02.544375 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 189361us
I20260806 04:08:02.456642 8403 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.552557 8485 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:02.411192 8292 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.408547 7994 consensus_queue.cc:1046] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.127s
I20260806 04:08:02.480208 8281 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.357239 8345 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.173s
W20260806 04:08:02.556855 4768 connection.cc:232] RPC call timeout handler was delayed by -1.2442940012551844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
W20260806 04:08:02.578891 4768 connection.cc:232] RPC call timeout handler was delayed by -1.188494001282379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
W20260806 04:08:02.579334 4768 connection.cc:232] RPC call timeout handler was delayed by -1.1776940012350678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260806 04:08:02.579715 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0948940012603998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
W20260806 04:08:02.580067 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0894940011203289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
W20260806 04:08:02.580415 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0696940012276173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
I20260806 04:08:02.581284 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:02.414582 8296 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:02.564452 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 279452us
W20260806 04:08:02.578395 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 96501.1us
I20260806 04:08:02.460929 8474 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 04:08:02.587881 4992 leader_election.cc:334] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.986s (SENT)
I20260806 04:08:02.588480 4992 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.456964 8295 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.414577 8094 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.140s
I20260806 04:08:02.589953 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.538681 8460 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:02.467219 8291 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.595328 5816 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:02.599858 4768 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.540467 8362 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.566246 8449 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.575415 8292 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:02.608652 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:02.603662 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 72242.6us
I20260806 04:08:02.558422 8458 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:02.560081 8192 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.561988 8348 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:02.466027 8242 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.606146 4814 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:02.591737 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.632182 4768 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:02.632467 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:02.577551 8267 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.570087 8403 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:02.592017 8345 consensus_queue.cc:1225] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer 99690415a7b3441fbdb5e943667556a4 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 04:08:02.649729 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.615723 8485 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:02.625943 5035 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:02.632591 4816 raft_consensus.cc:2396] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:08:02.642081 4805 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:02.466509 8486 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.614816 8460 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:02.626820 8362 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 04:08:02.666688 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 48304.3us
I20260806 04:08:02.668000 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.628474 8069 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.691818 4816 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 1.
I20260806 04:08:02.693184 4768 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.630868 8358 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:02.702589 4768 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:02.630505 8458 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:02.701350 5030 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:02.697762 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:02.667837 8449 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:02.669759 8339 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:02.643615 8354 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:02.700086 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 117183us
I20260806 04:08:02.641130 8292 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.648274 8348 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:02.754544 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:02.737452 5863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 639) took 3562 ms. Trace:
W20260806 04:08:02.711170 8490 log.cc:924] Time spent T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.155s user 0.000s sys 0.002s
I20260806 04:08:02.663400 8239 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:02.708706 4768 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:02.758211 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 49755.5us
I20260806 04:08:02.759644 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:02.747918 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:02.655929 8403 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:02.727991 8362 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:02.707554 8069 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:02.755936 5863 rpcz_store.cc:276] 0806 04:07:59.175091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.175288 (+ 197us) service_pool.cc:227] Handling call
0806 04:07:59.209848 (+ 34560us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:07:59.559938 (+350090us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.737088 (+3177150us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b540051dbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:02.737324 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":795191680}
I20260806 04:08:02.771618 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:02.741189 8358 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:08:02.708878 8449 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.745554 8489 consensus_queue.cc:1035] T cfd13a1778034c599358d37b5d72c374 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.300s
W20260806 04:08:02.995957 4924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1490) took 3575 ms (client timeout 1574 ms). Trace:
W20260806 04:08:03.004518 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 445862us
W20260806 04:08:03.014436 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 385195us
I20260806 04:08:02.734529 8458 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:03.038240 8094 consensus_queue.cc:1046] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:08:02.868981 5029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 1974) took 3274 ms (client timeout 2132 ms). Trace:
I20260806 04:08:02.778847 5859 raft_consensus.cc:2379] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:08:02.848371 4923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1485) took 3465 ms (client timeout 2665 ms). Trace:
I20260806 04:08:02.984272 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 366) took 1118 ms. Trace:
I20260806 04:08:03.030958 8494 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:03.034559 8180 consensus_queue.cc:1046] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:08:02.749970 8339 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:03.036561 8354 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:08:03.028182 4924 rpcz_store.cc:269] 0806 04:07:59.420406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.462029 (+ 41623us) service_pool.cc:227] Handling call
0806 04:08:02.995677 (+3533648us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400288054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:02.995871 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3388247296}
W20260806 04:08:02.906013 8473 log.cc:924] Time spent T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.255s user 0.000s sys 0.002s
W20260806 04:08:03.027668 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 256117us
W20260806 04:08:03.042697 4923 rpcz_store.cc:269] 0806 04:07:59.382980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.383753 (+ 773us) service_pool.cc:227] Handling call
0806 04:08:02.847977 (+3464224us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003552d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:02.848210 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2459376000}
W20260806 04:08:03.040999 5029 rpcz_store.cc:269] 0806 04:07:59.594337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.673282 (+ 78945us) service_pool.cc:227] Handling call
0806 04:08:02.868683 (+3195401us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b540037d054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:02.868883 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":422870272}
I20260806 04:08:03.047411 8382 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:02.768873 8140 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.669s user 0.007s sys 0.007s
I20260806 04:08:03.076375 8382 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:03.076690 8398 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 2.052s user 0.012s sys 0.003s
W20260806 04:08:02.947151 8283 consensus_meta.cc:287] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.649s user 0.016s sys 0.000s
I20260806 04:08:02.851814 8197 raft_consensus.cc:3032] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:08:03.076828 8140 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:02.833542 8289 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:02.908771 8389 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.060074 8282 raft_consensus.cc:3032] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:03.060258 8339 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.067090 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:02.754310 8348 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:02.923852 8120 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:02.867631 8493 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:08:02.939745 8292 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:02.986057 8403 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:03.083932 4768 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:02.854499 8291 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:03.039238 4882 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:03.124711 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 85748.8us
W20260806 04:08:02.925832 8478 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.251s user 0.001s sys 0.000s
I20260806 04:08:03.091495 8354 raft_consensus.cc:686] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:03.137197 5854 raft_consensus.cc:2379] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:03.127110 8261 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:03.135330 7994 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
W20260806 04:08:03.124007 8498 log.cc:924] Time spent T eaa370887c344934b64e0b3f59c6a3b0 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.128s user 0.000s sys 0.002s
I20260806 04:08:03.149101 8140 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:03.151985 4928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1495) took 3584 ms (client timeout 1976 ms). Trace:
W20260806 04:08:03.152463 4928 rpcz_store.cc:269] 0806 04:07:59.567365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.584797 (+ 17432us) service_pool.cc:227] Handling call
0806 04:08:03.151654 (+3566857us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54000f6c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:03.151897 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2250213248}
I20260806 04:08:03.156999 4814 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:08:03.158823 4925 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
W20260806 04:08:03.159956 8192 consensus_queue.cc:1159] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "99690415a7b3441fbdb5e943667556a4" responder_term: 7 status { last_received { term: 4 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: [6.5-6.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:08:03.163692 4928 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:03.163293 8200 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.165493 4928 raft_consensus.cc:1226] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:03.167107 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:03.167992 4805 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:03.075251 8494 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:03.168826 8493 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:03.169726 8493 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:03.171128 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1333) took 1264 ms. Trace:
I20260806 04:08:03.171695 4813 rpcz_store.cc:276] 0806 04:08:01.906849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.085477 (+178628us) service_pool.cc:227] Handling call
0806 04:08:02.085999 (+ 522us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:02.100161 (+ 14162us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:02.100204 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:02.100658 (+ 454us) log.cc:842] Serialized 26 byte log entry
0806 04:08:02.293599 (+192941us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:08:02.293639 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.293703 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:03.170804 (+877101us) raft_consensus.cc:1671] finished
0806 04:08:03.170826 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:03.171023 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3239,"raft.queue_time_us":153891,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":138,"thread_start_us":126521,"threads_started":1,"wal-append.queue_time_us":397021}
I20260806 04:08:03.168921 8494 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:03.127568 8283 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:03.184917 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 370) took 1279 ms. Trace:
I20260806 04:08:03.185475 8200 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:03.188241 8512 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260806 04:08:03.148479 7932 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:03.190420 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:03.190996 8354 consensus_queue.cc:237] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.042984 4808 rpcz_store.cc:276] 0806 04:08:01.865688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.058696 (+193008us) service_pool.cc:227] Handling call
0806 04:08:02.081720 (+ 23024us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.081954 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:08:02.201863 (+119909us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.267111 (+ 65248us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.575200 (+308089us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:02.575254 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:02.575315 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:02.575383 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:02.983991 (+408608us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b54003d8950. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:02.984172 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2332,"prepare.queue_time_us":157966,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":77,"raft.queue_time_us":66368,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":10721,"spinlock_wait_cycles":52175232,"thread_start_us":415190,"threads_started":3,"wal-append.queue_time_us":139354}
I20260806 04:08:03.056869 8180 raft_consensus.cc:3032] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:03.072184 8449 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.202907 8180 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:03.205476 8489 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:03.077237 8382 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.114527 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.235051 4923 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:02.839407 8488 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:03.115883 8351 consensus_meta.cc:287] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.863s user 0.013s sys 0.007s
I20260806 04:08:03.115679 8348 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.116565 8172 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:03.075650 8094 raft_consensus.cc:3032] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:02.841346 8507 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:08:03.121510 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.125237 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:03.107923 8282 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:03.122486 8120 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:03.086241 8197 raft_consensus.cc:729] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:03.057662 8069 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.088032 4992 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:03.185246 8283 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:03.185516 4809 rpcz_store.cc:276] 0806 04:08:01.905382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.085917 (+180535us) service_pool.cc:227] Handling call
0806 04:08:02.086222 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.095552 (+ 9330us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:02.095585 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:02.095625 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:02.095661 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.184689 (+1089028us) spinlock_profiling.cc:243] Waited 9.12 ms on lock 0x7b540039a8d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:03.184737 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:03.184856 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20986624}
I20260806 04:08:03.184938 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:02.921573 8461 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:03.208148 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:03.206561 4876 connection.cc:232] RPC call timeout handler was delayed by -2.0518534919247031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
I20260806 04:08:03.204874 8180 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:03.257890 8399 consensus_meta.cc:287] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.929s user 0.013s sys 0.007s
W20260806 04:08:03.250706 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 125600us
I20260806 04:08:03.272074 4809 raft_consensus.cc:1261] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:03.272204 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:03.254411 4805 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:03.234180 7994 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:03.261983 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 174027us
I20260806 04:08:03.261039 8507 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:03.237671 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 43593.6us
I20260806 04:08:03.265375 4808 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:03.246116 8094 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:03.266594 8463 consensus_meta.cc:287] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.900s user 0.005s sys 0.016s
I20260806 04:08:03.286921 8463 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:03.281844 8282 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.288234 8282 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:03.288409 8087 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:03.282509 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.290963 8172 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:03.291618 8094 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.292616 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.293383 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:03.293898 8094 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:03.294546 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.294873 8486 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:03.297422 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.298007 4923 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:03.256742 8068 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:03.278729 4813 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
W20260806 04:08:03.277241 4992 leader_election.cc:334] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 2.665s (SENT)
I20260806 04:08:03.304428 4992 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:03.264570 8449 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:03.303506 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.294395 5867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 307) took 1210 ms. Trace:
I20260806 04:08:03.295279 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:03.244127 8339 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:03.325939 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 48736.2us
I20260806 04:08:03.310405 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:03.341800 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:03.342903 7643 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:03.343740 7643 raft_consensus.cc:686] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:03.344149 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1619) took 3877 ms. Trace:
I20260806 04:08:03.344607 4920 rpcz_store.cc:276] 0806 04:07:59.466951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.625656 (+158705us) service_pool.cc:227] Handling call
0806 04:08:01.897876 (+2272220us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:01.898616 (+ 740us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.343896 (+1445280us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b54000f6750. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:03.344076 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5204106624}
I20260806 04:08:03.345300 7643 consensus_queue.cc:237] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.328104 5867 rpcz_store.cc:276] 0806 04:08:02.084239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.094036 (+ 9797us) service_pool.cc:227] Handling call
0806 04:08:03.293203 (+1199167us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:03.294319 (+ 1116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8747392}
I20260806 04:08:03.240025 8489 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.352062 4924 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:03.309590 4923 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:03.266040 8450 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:03.364413 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.347039 8463 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:03.278473 8291 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:03.357797 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.357842 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.386094 4928 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:03.386292 4924 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:03.361539 4768 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:03.387488 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.388899 4924 raft_consensus.cc:2396] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:03.389875 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.308485 8068 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.256007 8488 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.368747 8339 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:03.287393 8197 consensus_queue.cc:260] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.285140 7994 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.285606 8507 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:08:03.384611 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.001s sys 0.029s
I20260806 04:08:03.337956 4882 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:03.391005 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 364) took 2423 ms. Trace:
I20260806 04:08:03.391448 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 369) took 1518 ms. Trace:
I20260806 04:08:03.502929 4926 rpcz_store.cc:276] 0806 04:08:00.967790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.983881 (+1016091us) service_pool.cc:227] Handling call
0806 04:08:01.984322 (+ 441us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.087804 (+103482us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.390739 (+1302935us) spinlock_profiling.cc:243] Waited 89.8 ms on lock 0x7b54001b85d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:03.390940 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206457728}
I20260806 04:08:03.503098 4817 rpcz_store.cc:276] 0806 04:08:01.872814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.086228 (+213414us) service_pool.cc:227] Handling call
0806 04:08:02.086486 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.210587 (+124101us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:02.210632 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:02.210710 (+ 78us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:02.210754 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.391193 (+1180439us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540040f954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:03.391246 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:03.391390 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":284854528}
I20260806 04:08:03.391433 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.504660 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.505163 4924 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:03.346387 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.397770 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1629) took 1133 ms. Trace:
I20260806 04:08:03.359220 8224 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:03.337261 8369 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:03.401311 8334 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:03.403224 8502 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.164s
I20260806 04:08:03.241089 8351 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.393702 4814 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:03.393747 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.520030 4923 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:03.280180 8399 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:03.496438 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 107241us
I20260806 04:08:03.499933 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:03.503170 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.002s sys 0.030s
I20260806 04:08:03.397616 8450 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:03.398603 8291 raft_consensus.cc:686] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:03.506822 4925 rpcz_store.cc:276] 0806 04:08:02.264502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.479243 (+214741us) service_pool.cc:227] Handling call
0806 04:08:03.153411 (+674168us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:03.159391 (+ 5980us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.397510 (+238119us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54000f6c50. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:03.397691 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1548678016}
I20260806 04:08:03.494953 8303 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:03.509125 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:03.509902 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 172049us
I20260806 04:08:03.423621 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33546 (request call id 697) took 1551 ms. Trace:
W20260806 04:08:03.406560 8446 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.851s user 0.010s sys 0.010s
I20260806 04:08:03.368752 8069 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:03.406751 8297 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:03.385620 8463 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:03.645990 8291 consensus_queue.cc:237] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.659663 8436 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260806 04:08:03.665058 8180 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:03.673547 8351 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:03.623003 8112 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:03.679072 8112 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.680122 8399 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:03.524837 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 753214us
W20260806 04:08:03.511099 8460 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.809s user 0.017s sys 0.005s
I20260806 04:08:03.531152 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.533831 4924 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:03.688805 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.689692 4924 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:03.428184 8447 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:03.552727 4810 raft_consensus.cc:1226] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260806 04:08:03.691768 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.284711 8461 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.560232 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.534299 4804 rpcz_store.cc:276] 0806 04:08:01.855901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:01.856080 (+ 179us) service_pool.cc:227] Handling call
0806 04:08:03.392026 (+1535946us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400231254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d99607c5 00007f23d9958c1a 00007f23de89c6d9 00007f23d7f86184 00007f23d7f86112 00007f23d7f86041 00007f23d7f85fbc 00007f23d7f85272 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:03.407445 (+ 15419us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":499138944}
I20260806 04:08:03.692932 4928 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:03.553264 4768 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:03.558110 8303 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.577687 7994 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:03.530577 8450 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.523526 8224 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:03.534144 8489 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.585258 8488 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:03.529485 8502 consensus_queue.cc:1225] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:08:03.418386 8283 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.691637 8447 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.707325 8282 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:03.604230 8192 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:03.559443 8446 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.587539 8454 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:03.611601 8458 consensus_meta.cc:287] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.008s
W20260806 04:08:03.638598 5862 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.028s sys 0.077s
W20260806 04:08:03.746384 5862 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s user 0.028s sys 0.077s
I20260806 04:08:03.604799 8197 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:08:03.690948 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.749658 4924 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:03.750833 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.553232 4920 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:03.566952 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:03.698086 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 144969us
W20260806 04:08:03.700111 4657 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 086f8af9433f44aa8aa4fccbfa90eb02 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica df2f1e7811114b8a995f6771fb6a6c8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:08:03.752224 4657 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54508.9us
I20260806 04:08:03.752619 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:03.705252 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1334) took 1798 ms. Trace:
I20260806 04:08:03.753396 4810 rpcz_store.cc:276] 0806 04:08:01.907181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.086546 (+179365us) service_pool.cc:227] Handling call
0806 04:08:02.157282 (+ 70736us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:03.698681 (+1541399us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.704990 (+ 6309us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540011dd54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:03.705180 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3208499456}
I20260806 04:08:03.705518 5029 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:03.661183 8463 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:03.755991 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.756861 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.759011 5032 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:03.759264 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.760219 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 268) took 1012 ms. Trace:
I20260806 04:08:03.714171 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:03.721446 8484 consensus_queue.cc:1159] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260806 04:08:03.758405 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.760610 5032 rpcz_store.cc:276] 0806 04:08:02.747812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.747956 (+ 144us) service_pool.cc:227] Handling call
0806 04:08:02.776432 (+ 28476us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:03.759488 (+983056us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.759992 (+ 504us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b5400321ed4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:03.760150 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2259219712}
I20260806 04:08:03.760927 4928 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:03.766422 4920 raft_consensus.cc:2396] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:03.694674 8460 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.764019 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:03.764531 8446 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.743361 8192 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:03.770578 8494 consensus_meta.cc:287] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.536s user 0.006s sys 0.012s
I20260806 04:08:03.772128 4924 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260806 04:08:03.764097 8224 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.761129 8454 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:03.776746 8192 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.776937 8454 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.782896 4926 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
W20260806 04:08:03.806066 8382 consensus_meta.cc:287] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.565s user 0.016s sys 0.001s
I20260806 04:08:03.808982 8303 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.750200 8458 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:03.783237 8493 consensus_meta.cc:287] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.613s user 0.014s sys 0.005s
I20260806 04:08:03.771332 8494 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:03.806357 8486 consensus_meta.cc:287] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.009s
I20260806 04:08:03.827869 8158 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:03.831382 4926 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:03.834812 8494 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.823069 8458 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.808864 8382 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.828706 8158 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:03.844573 8460 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.759873 8461 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:03.862447 8158 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:03.824807 8493 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:03.865002 4924 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:03.867128 8382 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.851018 8447 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:03.879814 8507 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:03.940948 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 188357us
I20260806 04:08:03.607954 4882 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:03.951337 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 582) took 1280 ms. Trace:
I20260806 04:08:03.950105 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:03.957136 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.120s user 0.036s sys 0.075s
W20260806 04:08:03.959586 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.122s user 0.037s sys 0.075s
W20260806 04:08:03.960307 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.13209e+06us
I20260806 04:08:03.959232 5862 rpcz_store.cc:276] 0806 04:08:02.670644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.670836 (+ 192us) service_pool.cc:227] Handling call
0806 04:08:02.706013 (+ 35177us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.706267 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:08:02.769388 (+ 63121us) log.cc:842] Serialized 14 byte log entry
0806 04:08:03.950835 (+1181447us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:03.950913 (+ 78us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:03.950989 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.951100 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:03.951271 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":198952,"prepare.queue_time_us":72362,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"thread_start_us":975816,"threads_started":1}
I20260806 04:08:03.972126 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:03.973052 4816 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:03.974970 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:03.978796 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:03.979514 4813 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:03.980120 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:03.981451 4804 raft_consensus.cc:2396] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:08:03.995841 4924 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260806 04:08:04.007412 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 57414.1us
I20260806 04:08:04.008005 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:04.008333 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:04.009337 4809 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:04.011420 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:04.024101 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:04.024959 4805 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
W20260806 04:08:04.006517 8120 consensus_meta.cc:287] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.739s user 0.007s sys 0.010s
I20260806 04:08:04.047865 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:04.058281 5032 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:08:04.007495 8348 consensus_meta.cc:287] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.752s user 0.016s sys 0.003s
W20260806 04:08:03.983186 8087 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.694s user 0.017s sys 0.000s
I20260806 04:08:04.070228 4813 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:04.071964 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:03.968714 8192 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:04.072393 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:04.064875 4816 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
W20260806 04:08:04.056994 8200 consensus_meta.cc:287] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.871s user 0.006s sys 0.011s
W20260806 04:08:04.127307 4879 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.593s user 0.024s sys 0.103s
W20260806 04:08:04.945473 8283 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.095s user 0.006s sys 0.008s
W20260806 04:08:05.203054 4876 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.824s user 0.005s sys 0.005s
I20260806 04:08:04.156716 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:05.369144 8507 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:05.370302 8334 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:05.370934 8454 consensus_meta.cc:287] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.593s user 0.017s sys 0.012s
I20260806 04:08:05.371640 8334 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.380702 8461 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.381675 8461 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.350440 8267 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:05.383626 8267 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.386389 4818 raft_consensus.cc:2396] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:05.370328 8507 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:05.225597 5034 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.843s user 0.000s sys 0.015s
W20260806 04:08:04.385500 5030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.613s user 0.007s sys 0.028s
W20260806 04:08:05.225368 5026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.280s user 0.001s sys 0.035s
W20260806 04:08:04.545094 5038 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.714s user 0.004s sys 0.017s
W20260806 04:08:05.224414 4995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.408s user 0.000s sys 0.016s
W20260806 04:08:05.209561 4927 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.263s user 0.018s sys 0.034s
W20260806 04:08:04.172544 5813 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.374s user 0.005s sys 0.027s
W20260806 04:08:04.161801 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.534s user 0.057s sys 0.073s
I20260806 04:08:04.071085 8069 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:04.063212 8112 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:05.458292 8514 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.084s user 0.005s sys 0.001s
I20260806 04:08:04.070878 8158 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:05.458766 4770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.68:38873 timed out after 1.581s (SENT) [suppressed 18 similar messages]
I20260806 04:08:05.459666 8158 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.461081 8334 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:04.081717 8192 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:05.462486 8533 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.462802 8192 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:05.463555 8533 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.470806 8507 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.470347 8158 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:05.473353 8507 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:05.474298 5030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.668s user 0.007s sys 0.029s
I20260806 04:08:05.473510 8158 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:04.905767 8224 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.086s user 0.006s sys 0.005s
W20260806 04:08:05.477622 5038 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.618s user 0.006s sys 0.018s
W20260806 04:08:05.477916 8224 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.658s user 0.006s sys 0.005s
W20260806 04:08:04.844475 8242 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.784s user 0.016s sys 0.019s
W20260806 04:08:05.479163 8242 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.422s user 0.018s sys 0.020s
I20260806 04:08:04.072259 8281 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
W20260806 04:08:04.457468 8295 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.796s user 0.078s sys 0.165s
I20260806 04:08:04.071415 8299 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:04.072609 8339 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.481606 8299 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:04.879920 8346 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 4.238s user 0.024s sys 0.016s
W20260806 04:08:05.483182 8346 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 4.841s user 0.024s sys 0.017s
I20260806 04:08:04.083325 8348 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.485630 8281 raft_consensus.cc:686] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:05.485373 8069 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.487622 8348 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:05.487371 8281 consensus_queue.cc:237] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.488350 8069 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:04.073112 8219 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:04.071166 8372 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.515290 8219 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.516474 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 366) took 4485 ms. Trace:
W20260806 04:08:04.658604 8412 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.834s user 0.000s sys 0.019s
W20260806 04:08:05.517444 8412 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.704s user 0.000s sys 0.027s
I20260806 04:08:04.072827 8444 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.479775 8158 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.519981 8444 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.521565 8158 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:04.919349 8405 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.976s user 0.008s sys 0.007s
I20260806 04:08:04.048501 8450 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:05.522833 8405 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.579s user 0.008s sys 0.009s
I20260806 04:08:05.525643 8450 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:04.065999 8447 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:04.032972 8456 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.481881 8339 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.527166 8447 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.527336 8456 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.528801 5037 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:05.529644 5029 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:04.011857 8453 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:04.923236 8483 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.863s user 0.004s sys 0.012s
I20260806 04:08:04.071694 8486 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:04.032683 8488 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
W20260806 04:08:05.208554 4919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.189s user 0.000s sys 0.029s
W20260806 04:08:05.345480 4879 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.812s user 0.025s sys 0.104s
W20260806 04:08:05.263027 4651 catalog_manager.cc:4584] TS 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): ChangeConfig:REMOVE_PEER RPC failed for tablet 997e4d9ff3ca465199a888e8af23ed7b: Timed out: ChangeConfig RPC to 127.2.248.67:43701 timed out after 30.000s (SENT)
I20260806 04:08:05.348618 5812 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:05.341558 4876 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.962s user 0.006s sys 0.005s
W20260806 04:08:05.362126 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.74739e+06us
W20260806 04:08:05.363034 8291 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.427s user 0.005s sys 0.005s
I20260806 04:08:05.609627 8522 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:08:05.610105 8291 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.763s user 0.006s sys 0.005s
I20260806 04:08:05.610332 8522 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:08:05.364336 8068 consensus_meta.cc:287] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.866s user 0.011s sys 0.007s
I20260806 04:08:05.365777 8351 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.611024 8522 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:05.611079 8461 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:05.611776 8351 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.322525 8303 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:05.615262 8303 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:05.367107 8463 consensus_meta.cc:287] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.612s user 0.016s sys 0.007s
I20260806 04:08:05.617691 8463 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:05.367218 7643 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.763s user 0.002s sys 0.016s
I20260806 04:08:05.273373 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:05.368310 8283 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.508s user 0.007s sys 0.009s
I20260806 04:08:05.368407 8493 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:05.368537 4992 leader_election.cc:334] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.879s (SENT)
I20260806 04:08:05.322059 7932 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:05.371953 8454 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.374116 8391 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:08:05.408321 7714 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:05.409857 5036 raft_consensus.cc:2396] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260806 04:08:05.410760 5034 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.028s user 0.000s sys 0.016s
W20260806 04:08:05.410115 8180 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.744s user 0.009s sys 0.022s
W20260806 04:08:05.410297 4931 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.313s user 0.008s sys 0.024s
W20260806 04:08:05.439530 4995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.623s user 0.000s sys 0.017s
W20260806 04:08:05.448829 5026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.503s user 0.002s sys 0.035s
W20260806 04:08:05.445330 4927 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.499s user 0.023s sys 0.034s
W20260806 04:08:05.411123 8282 consensus_meta.cc:287] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.665s user 0.007s sys 0.011s
W20260806 04:08:05.446851 5028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.509s user 0.022s sys 0.008s
I20260806 04:08:05.421519 8267 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:04.070358 8120 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.476652 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 266) took 2729 ms. Trace:
W20260806 04:08:05.477170 5813 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.661s user 0.008s sys 0.037s
W20260806 04:08:04.873560 8354 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.328s user 0.000s sys 0.009s
W20260806 04:08:05.480370 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.826s user 0.058s sys 0.095s
I20260806 04:08:05.515944 8372 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:04.359361 8459 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.608s user 0.086s sys 0.180s
W20260806 04:08:05.480772 8295 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.854s user 0.079s sys 0.180s
W20260806 04:08:05.547549 8560 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.077s user 0.000s sys 0.002s
I20260806 04:08:05.516925 4930 rpcz_store.cc:276] 0806 04:08:01.031340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.048122 (+1016782us) service_pool.cc:227] Handling call
0806 04:08:02.067218 (+ 19096us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:02.068797 (+ 1579us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:08:02.238972 (+170175us) log.cc:842] Serialized 14 byte log entry
0806 04:08:03.032683 (+793711us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:03.033432 (+ 749us) log.cc:842] Serialized 26 byte log entry
0806 04:08:03.033663 (+ 231us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:03.033921 (+ 258us) log.cc:842] Serialized 14 byte log entry
0806 04:08:03.034107 (+ 186us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:03.034186 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:05.394817 (+2360631us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:05.515864 (+121047us) raft_consensus.cc:1671] finished
0806 04:08:05.515898 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:05.516198 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":291628,"prepare.queue_time_us":183793,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":77,"raft.queue_time_us":294548,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":12034,"spinlock_wait_cycles":146107648,"thread_start_us":603879,"threads_started":2,"wal-append.queue_time_us":1813678}
I20260806 04:08:05.473979 8507 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:05.583653 8483 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.525s user 0.005s sys 0.014s
I20260806 04:08:05.581976 8299 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:05.598806 4919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.580s user 0.000s sys 0.038s
I20260806 04:08:05.599426 8219 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:05.603365 8447 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:05.604332 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1340) took 3512 ms (client timeout 1969 ms). Trace:
W20260806 04:08:05.609182 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 261036us
I20260806 04:08:05.622071 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:05.722007 5032 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.625s user 0.010s sys 0.030s
I20260806 04:08:05.722506 5032 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2.
W20260806 04:08:05.723134 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 272) took 1738 ms (client timeout 2030 ms). Trace:
I20260806 04:08:05.723655 8391 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:05.623154 5035 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:05.725595 8120 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:05.720662 4818 rpcz_store.cc:269] 0806 04:08:02.091929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.092232 (+ 303us) service_pool.cc:227] Handling call
0806 04:08:05.603943 (+3511711us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b540011ec54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:05.604146 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4422729472}
I20260806 04:08:05.728242 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:05.728962 4818 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:05.623080 4651 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 364222us
I20260806 04:08:05.614504 8351 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:05.626693 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:05.630070 7643 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.119s user 0.002s sys 0.017s
W20260806 04:08:05.639300 5028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.787s user 0.023s sys 0.009s
W20260806 04:08:05.640926 4931 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.618s user 0.010s sys 0.024s
W20260806 04:08:05.649217 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 2.82783e+06us
I20260806 04:08:05.650810 8453 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:05.649746 4808 pending_rounds.cc:76] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 2
I20260806 04:08:05.748965 4809 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:05.749315 4808 pending_rounds.cc:97] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20260806 04:08:05.646230 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 2.26719e+06us
I20260806 04:08:05.639555 4992 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:05.639174 7932 raft_consensus.cc:686] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:05.637212 8533 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:05.643399 8444 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:05.655753 5036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2000) took 1578 ms (client timeout 1972 ms). Trace:
I20260806 04:08:05.672942 5037 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:05.675051 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:05.649041 8282 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:05.638206 8068 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.673367 4925 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:05.696636 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1630) took 3427 ms. Trace:
W20260806 04:08:05.692966 8197 consensus_meta.cc:287] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.845s user 0.009s sys 0.028s
W20260806 04:08:05.705998 8559 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260806 04:08:05.694510 7714 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:05.709247 8551 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.230s user 0.002s sys 0.002s
W20260806 04:08:05.709645 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.648s user 0.004s sys 0.046s
I20260806 04:08:05.658066 5030 rpcz_store.cc:276] 0806 04:08:02.747061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.748006 (+ 945us) service_pool.cc:227] Handling call
0806 04:08:02.748366 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.748639 (+ 273us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:08:02.769620 (+ 20981us) log.cc:842] Serialized 14 byte log entry
0806 04:08:05.475670 (+2706050us) log.cc:842] Serialized 14 byte log entry
0806 04:08:05.475898 (+ 228us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:05.475977 (+ 79us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:05.476060 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:05.476213 (+ 153us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:05.476521 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5879,"prepare.queue_time_us":49984,"prepare.run_cpu_time_us":174,"prepare.run_wall_time_us":6737,"raft.queue_time_us":109975,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":335,"thread_start_us":2702413,"threads_started":1,"wal-append.queue_time_us":2605823}
W20260806 04:08:05.702905 8561 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.098s user 0.001s sys 0.001s
I20260806 04:08:05.715433 8507 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:05.719523 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1627) took 3455 ms. Trace:
I20260806 04:08:05.720054 8372 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:05.723790 8486 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:05.723469 5032 rpcz_store.cc:269] 0806 04:08:03.984416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.984601 (+ 185us) service_pool.cc:227] Handling call
0806 04:08:05.723040 (+1738439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:05.739311 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:05.740082 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1628) took 3475 ms. Trace:
W20260806 04:08:05.740931 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 3.14541e+06us
I20260806 04:08:05.742906 8493 consensus_queue.cc:1046] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:08:05.744606 8461 consensus_queue.cc:1046] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260806 04:08:06.294157 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:06.191344 4770 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.008s sys 0.008s
W20260806 04:08:06.303658 4770 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.844s user 0.012s sys 0.008s
I20260806 04:08:05.880225 5035 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:05.888417 4818 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
W20260806 04:08:06.304240 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 907509us
W20260806 04:08:06.298895 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.010s sys 0.014s
W20260806 04:08:06.315495 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.949s user 0.012s sys 0.014s
W20260806 04:08:06.271315 4814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.848s user 0.015s sys 0.040s
W20260806 04:08:06.315932 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 949665us
W20260806 04:08:06.316942 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.267s user 0.014s sys 0.058s
I20260806 04:08:05.760639 4808 raft_consensus.cc:2866] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:08:06.134420 4817 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.010s sys 0.022s
W20260806 04:08:06.033198 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 62787.4us
I20260806 04:08:05.899737 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 254) took 5316 ms. Trace:
W20260806 04:08:05.758051 5036 rpcz_store.cc:269] 0806 04:08:04.067485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:04.075230 (+ 7745us) service_pool.cc:227] Handling call
0806 04:08:05.642472 (+1567242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:06.203403 5038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.005s sys 0.005s
W20260806 04:08:06.322764 5038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s user 0.006s sys 0.005s
W20260806 04:08:06.294109 8535 log.cc:924] Time spent T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.926s user 0.001s sys 0.003s
I20260806 04:08:06.295491 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:06.295202 8522 consensus_meta.cc:287] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.683s user 0.003s sys 0.013s
I20260806 04:08:06.325120 4923 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
W20260806 04:08:06.039853 8192 consensus_meta.cc:287] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.576s user 0.018s sys 0.000s
W20260806 04:08:05.868165 8459 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.969s user 0.092s sys 0.211s
W20260806 04:08:05.868011 8354 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.171s user 0.001s sys 0.009s
W20260806 04:08:06.076470 8544 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.318s user 0.001s sys 0.001s
W20260806 04:08:06.328508 4817 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s user 0.013s sys 0.027s
W20260806 04:08:06.329164 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 131740us
I20260806 04:08:05.745263 8303 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:06.334844 4814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.222s user 0.017s sys 0.047s
W20260806 04:08:06.321753 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2016) took 2232 ms (client timeout 1671 ms). Trace:
W20260806 04:08:06.338092 4920 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.010s sys 0.016s
I20260806 04:08:06.076680 8242 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:06.223729 8112 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:06.341395 4920 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.960s user 0.010s sys 0.017s
I20260806 04:08:06.341821 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:06.340764 5035 rpcz_store.cc:269] 0806 04:08:04.079718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:05.228438 (+1148720us) service_pool.cc:227] Handling call
0806 04:08:06.311993 (+1083555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:05.793663 8454 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:06.348147 4771 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.009s sys 0.027s
W20260806 04:08:06.348594 4771 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.013s user 0.009s sys 0.027s
I20260806 04:08:05.831743 8488 raft_consensus.cc:686] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:05.787309 8463 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:06.327006 8224 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:06.345182 4993 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.008s sys 0.016s
I20260806 04:08:06.301858 4927 rpcz_store.cc:276] 0806 04:08:02.268923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.944635 (+675712us) service_pool.cc:227] Handling call
0806 04:08:02.945112 (+ 477us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.945380 (+ 268us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:02.945689 (+ 309us) log.cc:842] Serialized 14 byte log entry
0806 04:08:05.683941 (+2738252us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:05.690848 (+ 6907us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:05.690989 (+ 141us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:05.691122 (+ 133us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:05.696558 (+ 5436us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10542,"prepare.queue_time_us":137,"prepare.run_cpu_time_us":2639,"prepare.run_wall_time_us":5183,"raft.queue_time_us":21514,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":20208,"thread_start_us":2687252,"threads_started":1,"wal-append.queue_time_us":2581493}
W20260806 04:08:06.351193 4993 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.996s user 0.008s sys 0.017s
I20260806 04:08:06.076997 8180 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.326148 8391 raft_consensus.cc:3032] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:06.302080 4931 rpcz_store.cc:276] 0806 04:08:02.264213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.360124 (+ 95911us) service_pool.cc:227] Handling call
0806 04:08:02.360607 (+ 483us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:02.560638 (+200031us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:02.560688 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:02.561294 (+ 606us) log.cc:842] Serialized 26 byte log entry
0806 04:08:02.660724 (+ 99430us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:02.696149 (+ 35425us) log.cc:842] Serialized 14 byte log entry
0806 04:08:03.004118 (+307969us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b18002db6e0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23dd802bcf 00007f23d6a327b6 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d
0806 04:08:05.739413 (+2735295us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:05.739583 (+ 170us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:05.739672 (+ 89us) raft_consensus.cc:1671] finished
0806 04:08:05.739786 (+ 114us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:05.740011 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":331921,"prepare.queue_time_us":35030,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":500471,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":9302,"spinlock_wait_cycles":10506496,"thread_start_us":2959081,"threads_started":3,"wal-append.queue_time_us":123066,"wal-append.run_cpu_time_us":4550,"wal-append.run_wall_time_us":1114650}
I20260806 04:08:05.788960 8334 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.557s
I20260806 04:08:05.970093 8267 consensus_queue.cc:1046] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.375s
I20260806 04:08:05.826875 8401 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:06.352650 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2005) took 2283 ms (client timeout 2202 ms). Trace:
I20260806 04:08:06.353621 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:06.301596 4919 rpcz_store.cc:276] 0806 04:08:02.263923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.342639 (+ 78716us) service_pool.cc:227] Handling call
0806 04:08:02.343093 (+ 454us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.343367 (+ 274us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:08:02.361684 (+ 18317us) log.cc:842] Serialized 14 byte log entry
0806 04:08:05.718991 (+3357307us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:05.719089 (+ 98us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:05.719178 (+ 89us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:05.719276 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:05.719447 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":388894,"prepare.queue_time_us":21521,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":396325,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":151,"thread_start_us":2970978,"threads_started":2,"wal-append.queue_time_us":360638,"wal-append.run_cpu_time_us":2622,"wal-append.run_wall_time_us":1123967}
I20260806 04:08:06.360512 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:06.359267 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 2.34503e+06us
W20260806 04:08:06.353535 4879 connection.cc:232] RPC call timeout handler was delayed by -1.9644220930058509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
I20260806 04:08:06.319985 5034 rpcz_store.cc:276] 0806 04:08:00.559682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.719129 (+159447us) service_pool.cc:227] Handling call
0806 04:08:00.719640 (+ 511us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:00.738270 (+ 18630us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:00.738308 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:00.763862 (+ 25554us) log.cc:842] Serialized 52 byte log entry
0806 04:08:00.979931 (+216069us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:03.381999 (+2402068us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b180042eec0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a2f155 00007f23dd32f167 00007f23d9964807 00007f23d9973f0f
0806 04:08:03.382166 (+ 167us) log.cc:842] Serialized 14 byte log entry
0806 04:08:05.701990 (+2319824us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:05.746313 (+ 44323us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:05.750905 (+ 4592us) raft_consensus.cc:1671] finished
0806 04:08:05.784851 (+ 33946us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:05.876318 (+ 91467us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18103,"prepare.queue_time_us":187343,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":18357,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":34404,"spinlock_wait_cycles":39006976,"thread_start_us":2723687,"threads_started":3,"wal-append.queue_time_us":2667860,"wal-append.run_cpu_time_us":3724,"wal-append.run_wall_time_us":1030650}
I20260806 04:08:06.363484 4818 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:06.363494 8531 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.003s sys 0.010s
I20260806 04:08:06.336313 8192 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.369964 8401 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.371456 8192 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.372181 4930 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:06.325119 8522 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:06.364420 4805 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.003s sys 0.007s
I20260806 04:08:06.362046 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:06.369395 8545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.771s user 0.010s sys 0.015s
W20260806 04:08:06.377125 8545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.982s user 0.010s sys 0.016s
I20260806 04:08:06.363933 8230 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:08:06.368062 4879 connection.cc:232] RPC call timeout handler was delayed by -1.8654011879116297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
W20260806 04:08:06.382664 4879 connection.cc:232] RPC call timeout handler was delayed by -1.5312327540013939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260806 04:08:06.366212 5032 rpcz_store.cc:269] 0806 04:08:04.068827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.352302 (+2283475us) service_pool.cc:214] Skipping call since client already timed out
0806 04:08:06.352530 (+ 228us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 04:08:06.379644 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:06.381033 4805 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.739s user 0.004s sys 0.007s
W20260806 04:08:06.369601 8531 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s user 0.004s sys 0.010s
I20260806 04:08:06.356231 8391 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:05.867583 7932 consensus_queue.cc:237] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.365309 8454 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.393690 8524 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:06.394829 8200 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.395694 8200 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.388253 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:06.401628 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2011) took 2319 ms (client timeout 2535 ms). Trace:
W20260806 04:08:06.402181 5035 rpcz_store.cc:269] 0806 04:08:04.070257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.355065 (+2284808us) service_pool.cc:227] Handling call
0806 04:08:06.389691 (+ 34626us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.394245 8399 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:06.366631 8282 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:06.385092 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.421375 4813 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:06.427331 5862 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:06.424635 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:06.401227 8281 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.801s user 0.046s sys 0.042s
I20260806 04:08:06.359730 8224 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:06.433852 8281 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s user 0.046s sys 0.046s
I20260806 04:08:06.441293 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:06.440953 5036 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:06.404492 8453 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:06.434353 8530 log.cc:924] Time spent T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.276s user 0.000s sys 0.001s
W20260806 04:08:06.451249 5036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2010) took 2374 ms (client timeout 2374 ms). Trace:
I20260806 04:08:06.399274 8192 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:08:06.451710 5036 rpcz_store.cc:269] 0806 04:08:04.069994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.361080 (+2291086us) service_pool.cc:227] Handling call
0806 04:08:06.443952 (+ 82872us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.426697 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.449839 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.454017 4924 raft_consensus.cc:2396] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260806 04:08:06.442713 4810 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:06.378742 8112 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.426332 8399 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:06.381541 8242 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.460253 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:06.432087 8534 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:06.463624 5035 raft_consensus.cc:2354] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6.
I20260806 04:08:06.464874 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:06.465512 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2008) took 2396 ms. Trace:
I20260806 04:08:06.407160 8180 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.466023 5035 rpcz_store.cc:276] 0806 04:08:04.069527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.425952 (+2356425us) service_pool.cc:227] Handling call
0806 04:08:06.465433 (+ 39481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.466925 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:06.460013 8483 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.045s sys 0.077s
I20260806 04:08:06.472244 5034 raft_consensus.cc:1799] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Handling vote request from an unknown peer 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:06.447566 8454 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.455127 8282 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.429409 8230 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.518080 5030 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:06.531268 5034 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7. Re-sending same reply.
W20260806 04:08:06.529671 8483 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.804s user 0.045s sys 0.077s
W20260806 04:08:06.562718 5030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2009) took 2490 ms (client timeout 2997 ms). Trace:
W20260806 04:08:06.564625 5030 rpcz_store.cc:269] 0806 04:08:04.069747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.373298 (+2303551us) service_pool.cc:227] Handling call
0806 04:08:06.559862 (+186564us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.553453 4879 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:06.563858 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2006) took 2491 ms (client timeout 3073 ms). Trace:
W20260806 04:08:06.566071 5034 rpcz_store.cc:269] 0806 04:08:04.069055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.376173 (+2307118us) service_pool.cc:227] Handling call
0806 04:08:06.560827 (+184654us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:06.563127 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2007) took 2492 ms (client timeout 3150 ms). Trace:
W20260806 04:08:06.567934 5032 rpcz_store.cc:269] 0806 04:08:04.069299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.408616 (+2339317us) service_pool.cc:227] Handling call
0806 04:08:06.561495 (+152879us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:06.542382 8577 log.cc:924] Time spent T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260806 04:08:06.544443 8180 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:06.541297 8522 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:06.540371 8507 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:06.534971 8200 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:06.547117 8486 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:06.567255 8068 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:06.542877 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:06.579309 8573 log.cc:924] Time spent T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260806 04:08:06.530809 8459 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260806 04:08:06.579759 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:06.569139 8192 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:06.590801 4770 connection.cc:232] RPC call timeout handler was delayed by -2.2024789769202471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
I20260806 04:08:06.468709 8488 consensus_queue.cc:237] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.599792 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.600260 8230 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.598839 4818 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:06.594177 5034 pending_rounds.cc:76] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Aborting all ops after (but not including) 4
I20260806 04:08:06.584199 8200 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:06.605392 5034 pending_rounds.cc:97] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.5
I20260806 04:08:06.605754 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.595458 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:06.617388 4809 consensus_meta.cc:287] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.868s user 0.013s sys 0.000s
I20260806 04:08:06.607331 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:06.567147 8534 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.617936 4809 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:06.621618 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 2183) took 1003 ms. Trace:
I20260806 04:08:06.621670 5032 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:08:06.621018 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.590283 5857 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:06.620751 5034 raft_consensus.cc:2894] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Aborting config change with OpId 2.5: Aborted: Op aborted by new leader
I20260806 04:08:06.624570 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2014) took 2544 ms. Trace:
I20260806 04:08:06.626547 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:06.583645 8391 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.622102 4809 rpcz_store.cc:276] 0806 04:08:05.618255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:05.626148 (+ 7893us) service_pool.cc:227] Handling call
0806 04:08:06.621559 (+995411us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.627182 5032 rpcz_store.cc:276] 0806 04:08:04.078561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.570254 (+2491693us) service_pool.cc:227] Handling call
0806 04:08:06.593707 (+ 23453us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:06.622203 (+ 28496us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.622406 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.606269 8339 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:06.624475 4770 connection.cc:232] RPC call timeout handler was delayed by -2.03786718682386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260806 04:08:06.630507 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:06.629484 8391 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:06.623975 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.634478 5859 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:06.626215 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.635085 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1510) took 3366 ms. Trace:
I20260806 04:08:06.635324 5863 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:06.636381 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:06.654443 4882 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:06.644225 8459 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:06.635545 4928 rpcz_store.cc:276] 0806 04:08:03.269050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.773314 (+504264us) service_pool.cc:227] Handling call
0806 04:08:03.775276 (+ 1962us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:05.285245 (+1509969us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:05.285323 (+ 78us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:05.328186 (+ 42863us) log.cc:842] Serialized 26 byte log entry
0806 04:08:05.919961 (+591775us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:08:05.920023 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:05.952690 (+ 32667us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:06.634726 (+682036us) raft_consensus.cc:1671] finished
0806 04:08:06.634750 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:06.634909 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1473576,"raft.queue_time_us":1544950,"raft.run_cpu_time_us":589,"raft.run_wall_time_us":594,"thread_start_us":462925,"threads_started":1,"wal-append.queue_time_us":1106512}
I20260806 04:08:06.674223 8242 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:06.677260 8532 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:06.678107 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:06.680486 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.680673 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2179) took 1064 ms. Trace:
I20260806 04:08:06.682193 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:06.683859 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:06.687255 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:06.665882 8589 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:06.687531 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.688714 4930 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:06.683053 8532 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:06.691251 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.691939 8532 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.692867 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.693398 8532 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:06.695166 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:06.697212 4809 raft_consensus.cc:1226] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:06.701182 8563 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.701824 8563 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.708597 5862 raft_consensus.cc:2396] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:06.710525 4813 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:06.716609 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.718719 4810 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:06.683688 4816 rpcz_store.cc:276] 0806 04:08:05.616359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:05.621390 (+ 5031us) service_pool.cc:227] Handling call
0806 04:08:06.631158 (+1009768us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:06.631428 (+ 270us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:06.632075 (+ 647us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:06.632472 (+ 397us) log.cc:842] Serialized 14 byte log entry
0806 04:08:06.671366 (+ 38894us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:06.674427 (+ 3061us) log.cc:842] Serialized 26 byte log entry
0806 04:08:06.676080 (+ 1653us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:06.676138 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.679504 (+ 3366us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:06.679552 (+ 48us) raft_consensus.cc:1671] finished
0806 04:08:06.680347 (+ 795us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:06.680481 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2961,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":1503,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":252,"spinlock_wait_cycles":4407168,"thread_start_us":31272,"threads_started":1,"wal-append.queue_time_us":31758}
I20260806 04:08:06.724807 5034 raft_consensus.cc:2866] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:06.717422 4926 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:08:06.687359 8282 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.687829 8112 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.728348 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2015) took 2640 ms. Trace:
I20260806 04:08:06.730631 5032 rpcz_store.cc:276] 0806 04:08:04.078874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.639325 (+2560451us) service_pool.cc:227] Handling call
0806 04:08:06.705415 (+ 66090us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:06.719097 (+ 13682us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:06.719135 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:06.719189 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:08:06.719238 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.719311 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:06.719460 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:06.703133 8589 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:06.728122 4882 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:06.733903 4813 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 04:08:06.734419 4813 pending_rounds.cc:76] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 6
I20260806 04:08:06.734742 4813 pending_rounds.cc:97] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 04:08:06.704653 8463 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.735054 4813 raft_consensus.cc:2866] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:06.712793 8354 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.753527 4930 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:08:06.750177 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:06.753587 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:06.757799 4919 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:06.736804 8589 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:06.760169 8463 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.760449 8289 consensus_queue.cc:1046] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:06.763242 4818 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:08:06.764065 4809 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 6. Ops: []
W20260806 04:08:06.766327 5816 connection.cc:232] RPC call timeout handler was delayed by -2.8729352080263197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
I20260806 04:08:06.748158 8282 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:06.771114 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34040 (request call id 638) took 32092 ms (client timeout 29999 ms). Trace:
I20260806 04:08:06.788095 4810 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: []
W20260806 04:08:06.788925 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 99033.3us
I20260806 04:08:06.741753 8459 raft_consensus.cc:686] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:06.780462 5034 rpcz_store.cc:269] 0806 04:07:34.679042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:34.679173 (+ 131us) service_pool.cc:227] Handling call
0806 04:07:34.712265 (+ 33092us) log.cc:842] Serialized 494 byte log entry
0806 04:08:06.770925 (+32058660us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12577,"raft.run_cpu_time_us":17575,"raft.run_wall_time_us":205621,"thread_start_us":45618,"threads_started":2,"wal-append.queue_time_us":257344,"wal-append.run_cpu_time_us":1799,"wal-append.run_wall_time_us":1006846}
I20260806 04:08:06.794956 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:06.762578 8463 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:06.800740 8592 log.cc:924] Time spent T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 04:08:06.798005 8281 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:06.800475 8589 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:06.747836 8565 consensus_queue.cc:1046] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260806 04:08:06.775369 8192 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.822481 4882 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:06.801898 8354 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.823017 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.802258 8242 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 04:08:06.830224 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 135535us
I20260806 04:08:06.837332 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.838929 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.958070 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.223994 4930 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:06.841764 5853 raft_consensus.cc:1226] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:08:06.950496 8112 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:06.830626 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.843968 8565 raft_consensus.cc:3032] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
W20260806 04:08:07.167690 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 461258us
I20260806 04:08:06.845831 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:06.836423 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1354) took 2844 ms. Trace:
I20260806 04:08:06.840420 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1512) took 3570 ms. Trace:
I20260806 04:08:06.848783 8354 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:06.934226 8589 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.850292 8192 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:06.837095 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.843120 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:06.856513 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.846558 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:06.848982 4816 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:06.845101 4813 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:06.837559 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:06.838339 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 285035us
I20260806 04:08:06.840806 5816 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:07.255478 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:06.855147 8463 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:06.943538 8614 raft_consensus.cc:3032] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
W20260806 04:08:06.847240 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54182.5us
I20260806 04:08:07.191083 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2181) took 1573 ms. Trace:
W20260806 04:08:06.939061 4925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1529) took 3572 ms (client timeout 3307 ms). Trace:
I20260806 04:08:06.838152 5860 raft_consensus.cc:1226] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:06.925138 4919 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:07.255956 4809 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:07.257552 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:06.951570 8563 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:06.847137 8609 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:06.978261 5854 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:07.249126 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:06.943871 8242 raft_consensus.cc:686] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:06.945535 8461 raft_consensus.cc:3032] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:06.937033 8200 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:07.245048 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:07.264856 5867 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:06.944321 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2184) took 1325 ms. Trace:
W20260806 04:08:07.258821 4768 connection.cc:232] RPC call timeout handler was delayed by -2.6684140430297703s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20260806 04:08:06.851517 8456 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:07.266822 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:07.269444 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:07.265484 8354 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:07.253208 4817 rpcz_store.cc:276] 0806 04:08:03.977149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.977298 (+ 149us) service_pool.cc:227] Handling call
0806 04:08:03.977689 (+ 391us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:04.035073 (+ 57384us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:04.035111 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:04.049434 (+ 14323us) log.cc:842] Serialized 26 byte log entry
0806 04:08:06.369179 (+2319745us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:06.710894 (+341715us) log.cc:842] Serialized 14 byte log entry
0806 04:08:06.787888 (+ 76994us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.796400 (+ 8512us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:06.803522 (+ 7122us) raft_consensus.cc:1671] finished
0806 04:08:06.803544 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:06.821713 (+ 18169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75899,"prepare.queue_time_us":152197,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":2458,"raft.queue_time_us":203513,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":220,"thread_start_us":1343645,"threads_started":3,"wal-append.queue_time_us":2326447}
W20260806 04:08:07.266183 4925 rpcz_store.cc:269] 0806 04:08:03.366577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.753508 (+386931us) service_pool.cc:227] Handling call
0806 04:08:03.755844 (+ 2336us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b58002d4b98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:06.938708 (+3182864us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b5400016154. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:06.938940 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4339419264}
I20260806 04:08:06.849059 8282 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.265621 8614 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:07.273317 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:07.272641 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:07.275105 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:07.276187 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:07.276424 4768 connection.cc:232] RPC call timeout handler was delayed by -2.5217140428721905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20260806 04:08:07.256109 5856 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:07.254163 8565 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:06.940097 8459 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:06.946458 8281 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.253608 4920 rpcz_store.cc:276] 0806 04:08:03.269619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.770404 (+500785us) service_pool.cc:227] Handling call
0806 04:08:03.770958 (+ 554us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:04.033717 (+262759us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:04.033760 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:04.056459 (+ 22699us) log.cc:842] Serialized 26 byte log entry
0806 04:08:05.380850 (+1324391us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b1800207b20. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a14345 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e1411 00007f23d992e5cf
0806 04:08:06.423210 (+1042360us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:06.423305 (+ 95us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.431992 (+ 8687us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:06.774822 (+342830us) spinlock_profiling.cc:243] Waited 7.65 ms on lock 0x7b54000c7ea0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d98e8771 00007f23d98f3220 00007f23d98f31c8 00007f23d98f3148 00007f23d98f3103 00007f23d98f238c 00007f23df963206 00007f23df95b808 00007f23d992f38b 00007f23d9931e80
0806 04:08:06.775407 (+ 585us) raft_consensus.cc:1671] finished
0806 04:08:06.775437 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:06.840360 (+ 64923us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":262271,"raft.queue_time_us":262546,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":13922,"spinlock_wait_cycles":54801024,"thread_start_us":965011,"threads_started":1,"wal-append.queue_time_us":2336000}
I20260806 04:08:07.265379 8617 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:07.267843 8192 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.260469 8112 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.272346 8391 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:07.280679 4930 raft_consensus.cc:2396] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:07.275652 4808 rpcz_store.cc:276] 0806 04:08:05.618638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:05.621808 (+ 3170us) service_pool.cc:227] Handling call
0806 04:08:05.622316 (+ 508us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:06.365324 (+743008us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:06.365372 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:06.375982 (+ 10610us) log.cc:842] Serialized 52 byte log entry
0806 04:08:06.730885 (+354903us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:06.730911 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.730958 (+ 47us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:06.944054 (+213096us) raft_consensus.cc:1671] finished
0806 04:08:06.944076 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:06.944246 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6397,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":32735,"thread_start_us":336884,"threads_started":1,"wal-append.queue_time_us":355419}
W20260806 04:08:07.459619 4768 connection.cc:232] RPC call timeout handler was delayed by -2.5190140430349857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
W20260806 04:08:07.324028 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 43911.6us
I20260806 04:08:07.277060 8405 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:07.372377 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.486461 8405 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:07.487476 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.487996 8405 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:07.488161 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.489600 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.274120 8461 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:07.460711 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 1.12043e+06us
I20260806 04:08:07.493180 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.387130 8158 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:07.460929 8578 consensus_queue.cc:1046] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 04:08:07.461794 4995 connection.cc:232] RPC call timeout handler was delayed by -1.8419618350453675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260806 04:08:07.506928 5812 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:07.463343 8520 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:07.464020 8192 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:07.281109 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:07.342276 4817 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:08:07.413337 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.467497 5860 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:08:07.468142 4810 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:07.468657 4919 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:07.469800 8112 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:07.323850 8609 raft_consensus.cc:686] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:07.475432 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:07.514382 5867 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:07.265960 4805 rpcz_store.cc:276] 0806 04:08:05.617300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:05.621298 (+ 3998us) service_pool.cc:227] Handling call
0806 04:08:05.622068 (+ 770us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:05.622567 (+ 499us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:06.388998 (+766431us) log.cc:842] Serialized 14 byte log entry
0806 04:08:06.784472 (+395474us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.126221 (+341749us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:07.126323 (+ 102us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:07.126390 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.126484 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.190980 (+ 64496us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11246,"prepare.queue_time_us":834614,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":386,"raft.queue_time_us":169504,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":24738,"spinlock_wait_cycles":102400,"thread_start_us":1409453,"threads_started":4,"wal-append.queue_time_us":332127}
I20260806 04:08:07.341583 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 273) took 3334 ms. Trace:
W20260806 04:08:07.314851 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 474128us
W20260806 04:08:07.481204 4768 connection.cc:232] RPC call timeout handler was delayed by -2.2787140430882573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20400000000000001s for a response
W20260806 04:08:07.516872 4768 connection.cc:232] RPC call timeout handler was delayed by -2.2409140430390835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20260806 04:08:07.484151 8614 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.485646 8282 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:07.486907 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 212413us
I20260806 04:08:07.344957 7714 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:07.291102 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:07.522441 4813 raft_consensus.cc:2396] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 8 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:07.524803 4879 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:07.275208 8463 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.309038 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.493026 4882 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:07.371663 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 2.03893e+06us
I20260806 04:08:07.491978 8461 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:07.386222 8608 log.cc:924] Time spent T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20260806 04:08:07.492928 4996 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:07.280287 8527 consensus_queue.cc:1046] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.213s
I20260806 04:08:07.510111 4818 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:07.534765 4927 raft_consensus.cc:2396] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:07.512280 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.515699 5029 rpcz_store.cc:276] 0806 04:08:04.006790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:04.009962 (+ 3172us) service_pool.cc:227] Handling call
0806 04:08:04.075159 (+ 65197us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:04.075330 (+ 171us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:05.530169 (+1454839us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.340620 (+1810451us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400017554. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:07.341501 (+ 881us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3347952384}
I20260806 04:08:07.520498 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.514072 8391 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:07.508723 8532 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260806 04:08:07.370076 8354 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.539343 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1540) took 1172 ms. Trace:
I20260806 04:08:07.543440 4927 rpcz_store.cc:276] 0806 04:08:06.366970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.412948 (+1045978us) service_pool.cc:227] Handling call
0806 04:08:07.539277 (+126329us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:07.542848 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 61799.7us
I20260806 04:08:07.544991 8302 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:07.545256 8164 raft_consensus.cc:3032] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:07.545655 8164 raft_consensus.cc:729] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:07.545967 8527 raft_consensus.cc:3032] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:07.546465 8527 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:07.546690 4995 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:07.539016 8520 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.369771 8242 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.518823 8617 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.548134 7714 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:07.549144 4920 raft_consensus.cc:1226] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 04:08:07.543366 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 1987) took 7944 ms. Trace:
I20260806 04:08:07.633618 4925 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:07.646977 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 100452us
I20260806 04:08:07.637867 4930 raft_consensus.cc:1261] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 04:08:07.647749 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 120613us
W20260806 04:08:07.648458 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 104562us
I20260806 04:08:07.532707 8614 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:07.649480 4928 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
W20260806 04:08:07.636423 8608 log.cc:924] Time spent T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260806 04:08:07.594368 8530 log.cc:924] Time spent T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260806 04:08:07.645530 8577 log.cc:924] Time spent T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
W20260806 04:08:07.657195 8564 log.cc:924] Time spent T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 1.133s user 0.005s sys 0.008s
I20260806 04:08:07.681496 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:07.681924 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1351) took 3711 ms. Trace:
I20260806 04:08:07.652571 8565 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.682332 4814 rpcz_store.cc:276] 0806 04:08:03.970248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.970420 (+ 172us) service_pool.cc:227] Handling call
0806 04:08:03.970900 (+ 480us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:04.026283 (+ 55383us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:04.026329 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:04.048503 (+ 22174us) log.cc:842] Serialized 26 byte log entry
0806 04:08:06.385753 (+2337250us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:06.661857 (+276104us) log.cc:842] Serialized 14 byte log entry
0806 04:08:06.663865 (+ 2008us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:06.665468 (+ 1603us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.546966 (+881498us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:07.681598 (+134632us) raft_consensus.cc:1671] finished
0806 04:08:07.681725 (+ 127us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.681866 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54872,"prepare.queue_time_us":179147,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":63936,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":20981,"spinlock_wait_cycles":12262784,"thread_start_us":2438041,"threads_started":2,"wal-append.queue_time_us":2335082}
I20260806 04:08:07.681313 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2012) took 3610 ms. Trace:
I20260806 04:08:07.633570 8629 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:07.688388 8520 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:07.567729 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 75016us
I20260806 04:08:07.542845 8532 consensus_queue.cc:1225] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer 22fe557f852d4415a96fd683f970cc24 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:08:07.669266 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:07.670192 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 269) took 4915 ms. Trace:
I20260806 04:08:07.751124 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2017) took 3668 ms. Trace:
I20260806 04:08:07.670073 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:07.670583 8391 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.672685 8461 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:07.673275 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:07.656258 8158 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:08:07.675797 8463 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:07.676437 8617 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:07.545678 8302 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.677950 8405 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.656833 8609 consensus_queue.cc:237] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.656980 8527 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.546802 8164 consensus_queue.cc:260] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.591919 8112 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:07.687569 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:07.651525 5028 rpcz_store.cc:276] 0806 04:07:59.599423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:59.873700 (+274277us) service_pool.cc:227] Handling call
0806 04:07:59.874534 (+ 834us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:00.034059 (+159525us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:08:00.053303 (+ 19244us) log.cc:842] Serialized 14 byte log entry
0806 04:08:02.045729 (+1992426us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b180048ff40. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32764 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:02.852631 (+806902us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b180048ff40. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a14345 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e157b 00007f23d996488a
0806 04:08:06.010103 (+3157472us) spinlock_profiling.cc:243] Waited 6.85 ms on lock 0x7b180048ff40. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e157b
0806 04:08:07.385980 (+1375877us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:07.386810 (+ 830us) log.cc:842] Serialized 26 byte log entry
0806 04:08:07.466774 (+ 79964us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:07.523363 (+ 56589us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.524911 (+ 1548us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.534192 (+ 9281us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.540511 (+ 6319us) raft_consensus.cc:1671] finished
0806 04:08:07.540531 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.543303 (+ 2772us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1366503,"prepare.queue_time_us":19639,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":75680,"raft.queue_time_us":1707567,"raft.run_cpu_time_us":435,"raft.run_wall_time_us":19411,"spinlock_wait_cycles":34129280,"thread_start_us":3058017,"threads_started":3,"wal-append.queue_time_us":5570776}
I20260806 04:08:07.591722 8578 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:07.735170 8454 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:07.665339 8483 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260806 04:08:07.736390 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2013) took 3658 ms. Trace:
I20260806 04:08:07.684948 8399 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.742424 4919 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:07.751307 5038 rpcz_store.cc:276] 0806 04:08:02.754526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:02.756193 (+ 1667us) service_pool.cc:227] Handling call
0806 04:08:02.799725 (+ 43532us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:02.816309 (+ 16584us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:03.375037 (+558728us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b18004229e0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32764 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:05.478424 (+2103387us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.461259 (+1982835us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:07.539621 (+ 78362us) log.cc:842] Serialized 26 byte log entry
0806 04:08:07.539844 (+ 223us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:07.539888 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.544535 (+ 4647us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.669880 (+125345us) raft_consensus.cc:1671] finished
0806 04:08:07.669999 (+ 119us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.670136 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1117110,"prepare.queue_time_us":21283,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":8591,"raft.queue_time_us":1160757,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":12345,"spinlock_wait_cycles":6834432,"thread_start_us":3491710,"threads_started":2,"wal-append.queue_time_us":913693}
W20260806 04:08:07.745502 4923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1525) took 4389 ms (client timeout 2739 ms). Trace:
I20260806 04:08:07.685166 5030 rpcz_store.cc:276] 0806 04:08:04.070540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.565392 (+2494852us) service_pool.cc:227] Handling call
0806 04:08:06.586879 (+ 21487us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:07.470506 (+883627us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.470536 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:07.482593 (+ 12057us) log.cc:842] Serialized 52 byte log entry
0806 04:08:07.646276 (+163683us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:07.646318 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.646388 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.681056 (+ 34668us) raft_consensus.cc:1671] finished
0806 04:08:07.681078 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.681229 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":876925,"raft.queue_time_us":877052,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":78760,"threads_started":1,"wal-append.queue_time_us":196075}
I20260806 04:08:07.767047 4995 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:07.914880 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 389) took 1162 ms. Trace:
W20260806 04:08:07.898720 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 243173us
I20260806 04:08:07.851390 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2004) took 3782 ms. Trace:
W20260806 04:08:07.805950 4992 connection.cc:232] RPC call timeout handler was delayed by -3.3642871507909149s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
I20260806 04:08:07.929045 4928 rpcz_store.cc:276] 0806 04:08:06.752376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.759422 (+ 7046us) service_pool.cc:227] Handling call
0806 04:08:06.772006 (+ 12584us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:07.657972 (+885966us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.914599 (+256627us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b540018fbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:07.914791 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2017518464}
I20260806 04:08:07.828902 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:45392 (request call id 441) took 1006 ms. Trace:
I20260806 04:08:07.933945 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1642) took 1164 ms. Trace:
I20260806 04:08:07.744557 4818 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
W20260806 04:08:07.928020 4923 rpcz_store.cc:269] 0806 04:08:03.355856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:03.691484 (+335628us) service_pool.cc:227] Handling call
0806 04:08:07.745266 (+4053782us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b5400354dd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:07.745428 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6046083840}
I20260806 04:08:07.907752 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 258) took 6951 ms. Trace:
I20260806 04:08:07.913203 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33546 (request call id 698) took 3893 ms. Trace:
I20260806 04:08:07.928319 5859 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:07.931329 4925 rpcz_store.cc:276] 0806 04:08:06.947682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.740536 (+792854us) service_pool.cc:227] Handling call
0806 04:08:07.828762 (+ 88226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:07.934432 4919 rpcz_store.cc:276] 0806 04:08:06.769484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.273122 (+503638us) service_pool.cc:227] Handling call
0806 04:08:07.933858 (+660736us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:07.917219 4770 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Timed out: RequestConsensusVote RPC to 127.2.248.68:38873 timed out after 1.581s (SENT)
I20260806 04:08:07.903319 8354 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:07.943053 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 195110us
I20260806 04:08:07.936136 4804 rpcz_store.cc:276] 0806 04:08:04.019561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:04.019755 (+ 194us) service_pool.cc:227] Handling call
0806 04:08:07.912659 (+3892904us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b5400231254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d99607c5 00007f23d9958c1a 00007f23de89c6d9 00007f23d7f86184 00007f23d7f86112 00007f23d7f86041 00007f23d7f85fbc 00007f23d7f85272 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:07.912999 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6080219008}
I20260806 04:08:07.943876 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1652) took 1149 ms. Trace:
I20260806 04:08:07.747499 8629 raft_consensus.cc:686] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:07.869005 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:07.944572 4920 rpcz_store.cc:276] 0806 04:08:06.774504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.667845 (+893341us) service_pool.cc:227] Handling call
0806 04:08:07.668163 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:07.851217 (+183054us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.851246 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:07.851286 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:07.851324 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.851384 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.924122 (+ 72738us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15784,"raft.queue_time_us":44278,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":27286}
I20260806 04:08:07.949052 5867 raft_consensus.cc:2354] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5.
W20260806 04:08:07.949337 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 199628us
I20260806 04:08:07.929553 5035 rpcz_store.cc:276] 0806 04:08:04.068595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.517530 (+2448935us) service_pool.cc:227] Handling call
0806 04:08:06.581730 (+ 64200us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:07.464444 (+882714us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.464481 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:07.464926 (+ 445us) log.cc:842] Serialized 26 byte log entry
0806 04:08:07.520923 (+ 55997us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:07.822768 (+301845us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.851082 (+ 28314us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.851158 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.851198 (+ 40us) raft_consensus.cc:1671] finished
0806 04:08:07.851208 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.851330 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":870226,"prepare.queue_time_us":157096,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":903147,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":12868,"spinlock_wait_cycles":1516416,"thread_start_us":294427,"threads_started":2,"wal-append.queue_time_us":74108}
I20260806 04:08:07.949731 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2003) took 3881 ms. Trace:
I20260806 04:08:07.928556 4931 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:07.951416 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 387) took 1200 ms. Trace:
I20260806 04:08:07.951838 4931 rpcz_store.cc:276] 0806 04:08:06.751516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.687196 (+935680us) service_pool.cc:227] Handling call
0806 04:08:07.951338 (+264142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:07.945755 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1650) took 1155 ms. Trace:
I20260806 04:08:07.953150 4927 rpcz_store.cc:276] 0806 04:08:06.773925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.665939 (+892014us) service_pool.cc:227] Handling call
0806 04:08:07.688899 (+ 22960us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:07.928858 (+239959us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.928891 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:07.928928 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:07.928972 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.929034 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.929140 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47564,"raft.queue_time_us":60896,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":39351,"thread_start_us":175585,"threads_started":1}
I20260806 04:08:07.931115 5034 rpcz_store.cc:276] 0806 04:08:04.078186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.567435 (+2489249us) service_pool.cc:227] Handling call
0806 04:08:06.587975 (+ 20540us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:07.467221 (+879246us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.467265 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:07.467899 (+ 634us) log.cc:842] Serialized 52 byte log entry
0806 04:08:07.515160 (+ 47261us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:07.654264 (+139104us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.665236 (+ 10972us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.665417 (+ 181us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.665483 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.690735 (+ 25252us) raft_consensus.cc:1671] finished
0806 04:08:07.690762 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.736318 (+ 45556us) inbound_call.cc:173] Queueing success response
0806 04:08:07.916779 (+180461us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b18001f3ae0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32638 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:07.919611 (+ 2832us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b18001f3ae0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32764 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
Metrics: {"mutex_wait_us":533402,"prepare.queue_time_us":186213,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":523545,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":6007,"spinlock_wait_cycles":12093440,"thread_start_us":176386,"threads_started":2,"wal-append.queue_time_us":187884}
I20260806 04:08:07.949553 5816 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:07.955821 8527 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:07.956878 5816 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:07.958884 8405 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:07.959540 8405 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:08:07.960197 8405 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:07.961535 4657 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 086f8af9433f44aa8aa4fccbfa90eb02 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica df2f1e7811114b8a995f6771fb6a6c8a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:08:07.962785 4992 leader_election.cc:334] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.733s (SENT)
I20260806 04:08:07.950167 5036 rpcz_store.cc:276] 0806 04:08:04.068326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.465608 (+2397282us) service_pool.cc:227] Handling call
0806 04:08:06.585415 (+119807us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:07.464053 (+878638us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.464079 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:07.464707 (+ 628us) log.cc:842] Serialized 52 byte log entry
0806 04:08:07.528701 (+ 63994us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:07.831077 (+302376us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.937712 (+106635us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.949427 (+ 11715us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.949501 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.949540 (+ 39us) raft_consensus.cc:1671] finished
0806 04:08:07.949549 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.949667 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":936567,"prepare.queue_time_us":157564,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":87,"raft.queue_time_us":891753,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":7439,"thread_start_us":262207,"threads_started":2,"wal-append.queue_time_us":185850}
I20260806 04:08:07.963274 4992 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:07.963933 7932 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:08:07.905287 8493 raft_consensus.cc:3032] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 04:08:07.967604 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 304305us
I20260806 04:08:07.918570 5854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 316) took 1227 ms. Trace:
I20260806 04:08:07.970198 8164 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:07.970441 5854 rpcz_store.cc:276] 0806 04:08:06.691222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.694838 (+ 3616us) service_pool.cc:227] Handling call
0806 04:08:07.918310 (+1223472us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540059ecd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:07.918464 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340348672}
I20260806 04:08:07.910359 8401 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:07.973884 8578 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:07.974349 8578 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:07.975289 8399 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:07.965571 8112 raft_consensus.cc:686] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:07.966884 8493 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:07.979568 8112 consensus_queue.cc:237] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:07.980638 8493 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:07.981737 8493 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:07.949344 8623 log.cc:924] Time spent T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.183s user 0.005s sys 0.004s
I20260806 04:08:07.968706 8486 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:07.748153 8566 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:08.000355 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 133137us
I20260806 04:08:08.001792 5816 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:07.935757 5026 rpcz_store.cc:276] 0806 04:08:00.955873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:00.997874 (+ 42001us) service_pool.cc:227] Handling call
0806 04:08:02.869168 (+1871294us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:02.869463 (+ 295us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:02.945314 (+ 75851us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.370367 (+4425053us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b18000bb5a0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23d99523d0 00007f23d9964576 00007f23d99750d7 00007f23d9975058
0806 04:08:07.370630 (+ 263us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:07.370717 (+ 87us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:07.370790 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.370907 (+ 117us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.907514 (+536607us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b540037d050. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:07.907676 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6829,"prepare.queue_time_us":76980,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":79,"raft.queue_time_us":1719976,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"spinlock_wait_cycles":4306654208,"thread_start_us":2690078,"threads_started":1,"wal-append.queue_time_us":2711550,"wal-append.run_cpu_time_us":18560,"wal-append.run_wall_time_us":1689637}
I20260806 04:08:08.003871 4995 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:07.989993 8486 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.012253 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.013186 4809 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:08.017266 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:08.017441 8405 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260806 04:08:08.018262 8486 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:08.017931 4813 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:08.029176 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.027402 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:07.994117 8447 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:08.020381 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 593) took 1487 ms. Trace:
W20260806 04:08:08.027546 4876 connection.cc:232] RPC call timeout handler was delayed by -2.5852359738200903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
I20260806 04:08:08.009725 8631 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
I20260806 04:08:08.044075 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 378) took 1419 ms. Trace:
I20260806 04:08:08.044603 4816 rpcz_store.cc:276] 0806 04:08:06.625090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.733009 (+107919us) service_pool.cc:227] Handling call
0806 04:08:06.758875 (+ 25866us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:07.270048 (+511173us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:08.043729 (+773681us) spinlock_profiling.cc:243] Waited 89.9 ms on lock 0x7b54000d07d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:08.043993 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206737280}
W20260806 04:08:08.042143 4876 connection.cc:232] RPC call timeout handler was delayed by -2.5360068851150572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
I20260806 04:08:08.046968 8120 raft_consensus.cc:3032] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:08.047523 8120 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:08.047601 5037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2002) took 3979 ms (client timeout 2146 ms). Trace:
W20260806 04:08:08.048260 5037 rpcz_store.cc:269] 0806 04:08:04.068056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:04.072125 (+ 4069us) service_pool.cc:227] Handling call
0806 04:08:08.047266 (+3975141us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b540032e6d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:08.047481 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":580142336}
I20260806 04:08:08.048995 8631 raft_consensus.cc:3032] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:08.049504 8631 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:08.036379 8158 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:08.054239 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 50489.5us
W20260806 04:08:08.046581 4876 connection.cc:232] RPC call timeout handler was delayed by -2.4286359739489853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2146s for a response
W20260806 04:08:08.055346 4876 connection.cc:232] RPC call timeout handler was delayed by -2.3782359738834202s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260806 04:08:08.055557 4995 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:08.055883 4876 connection.cc:232] RPC call timeout handler was delayed by -2.2576359738595784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2336s for a response
W20260806 04:08:08.056491 4876 connection.cc:232] RPC call timeout handler was delayed by -2.2234359737485647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23740000000000003s for a response
W20260806 04:08:08.041188 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 78450.5us
I20260806 04:08:08.038981 4814 raft_consensus.cc:2396] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:08.009368 7932 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.019186 8405 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:08.061944 4876 connection.cc:232] RPC call timeout handler was delayed by -2.0785359737928957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2535s for a response
W20260806 04:08:08.067936 4876 connection.cc:232] RPC call timeout handler was delayed by -1.6627359737176448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2997s for a response
W20260806 04:08:08.068315 4876 connection.cc:232] RPC call timeout handler was delayed by -1.5943359737284482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3073s for a response
W20260806 04:08:08.068652 4876 connection.cc:232] RPC call timeout handler was delayed by -1.5250359738711268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.315s for a response
W20260806 04:08:08.068993 4876 connection.cc:232] RPC call timeout handler was delayed by -1.4386359737254679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3246s for a response
W20260806 04:08:08.066344 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 260110us
I20260806 04:08:08.066988 4817 raft_consensus.cc:1261] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:08.040567 5862 rpcz_store.cc:276] 0806 04:08:06.557565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.562586 (+ 5021us) service_pool.cc:227] Handling call
0806 04:08:06.692748 (+130162us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:08.011002 (+1318254us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b540063f6d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:08.020291 (+ 9289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28846208}
W20260806 04:08:08.077546 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 75842.3us
I20260806 04:08:08.072815 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.076695 4879 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:08.002806 8532 consensus_queue.cc:1046] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.065s
W20260806 04:08:08.088387 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 93050.7us
I20260806 04:08:08.093658 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.123747 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:08.094568 4814 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:08:07.996155 8566 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.125720 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.127347 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.075613 8158 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:08.060613 8644 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:08.123569 8640 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:08.087689 8520 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:08.144708 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.090718 8454 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:08.146591 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 66297us
I20260806 04:08:07.751566 5032 rpcz_store.cc:276] 0806 04:08:04.080065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.736980 (+2656915us) service_pool.cc:227] Handling call
0806 04:08:06.767380 (+ 30400us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:07.465548 (+698168us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:07.465585 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:07.466203 (+ 618us) log.cc:842] Serialized 26 byte log entry
0806 04:08:07.530000 (+ 63797us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:07.747738 (+217738us) log.cc:842] Serialized 14 byte log entry
0806 04:08:07.748004 (+ 266us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:07.748110 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:07.748153 (+ 43us) raft_consensus.cc:1671] finished
0806 04:08:07.748162 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:07.748293 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":800177,"prepare.queue_time_us":148645,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":701333,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":5107,"spinlock_wait_cycles":949504,"thread_start_us":146152,"threads_started":2,"wal-append.queue_time_us":101524}
I20260806 04:08:08.150916 8401 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:08.151262 5856 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:08.151460 8401 raft_consensus.cc:686] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:08.152467 8401 consensus_queue.cc:237] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.152940 8454 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.155889 8454 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:08.140884 5854 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:08:08.141192 8158 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.160815 8158 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:08:08.162370 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 81015.6us
I20260806 04:08:08.057796 8456 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260806 04:08:08.144196 4817 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:08.143630 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:08.180552 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 137640us
I20260806 04:08:08.081816 7932 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.184924 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:08.194064 4810 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:08:08.145195 8640 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:08.199379 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.200435 4804 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:08.203886 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:08.203394 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.176502 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.213171 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.184486 8520 raft_consensus.cc:686] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
W20260806 04:08:08.217360 4876 leader_election.cc:334] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.202s (SENT)
I20260806 04:08:08.197510 8242 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.221920 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.091536 8629 consensus_queue.cc:237] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.196898 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:08.219312 4804 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:08.203209 8640 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:08.180267 8578 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.228324 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.184430 8197 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.037s
I20260806 04:08:08.191618 8600 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260806 04:08:08.231645 4817 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:08.141582 8631 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.233795 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:08.218562 8289 raft_consensus.cc:2741] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260806 04:08:08.217016 8399 consensus_queue.cc:1046] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260806 04:08:08.240545 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:08.241382 5853 raft_consensus.cc:2362] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
I20260806 04:08:08.240581 8399 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:08.249300 8399 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:08.196666 8644 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.250216 8399 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:08.250991 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 67067.6us
I20260806 04:08:08.216712 8463 consensus_queue.cc:1046] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260806 04:08:08.251978 8399 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:08.252552 8463 raft_consensus.cc:3032] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:08:08.252988 8463 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:08.253885 8463 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.254848 8453 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:08.254900 8520 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.256035 8453 consensus_queue.cc:1046] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:08:08.257794 8565 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:08.261926 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 382) took 1544 ms. Trace:
I20260806 04:08:08.231817 8180 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:08:08.268685 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.269461 4817 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:08.270416 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.261566 4770 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.263787 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:08.267385 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 50057.7us
I20260806 04:08:08.231972 8282 consensus_queue.cc:1046] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260806 04:08:08.268786 5857 raft_consensus.cc:2379] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:08.275137 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.264312 4924 rpcz_store.cc:276] 0806 04:08:06.717744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.726906 (+ 9162us) service_pool.cc:227] Handling call
0806 04:08:06.776091 (+ 49185us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:06.799243 (+ 23152us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:06.799282 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:06.838939 (+ 39657us) log.cc:842] Serialized 26 byte log entry
0806 04:08:08.052186 (+1213247us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b18002db7a0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a14345 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e1411 00007f23d992e5cf
0806 04:08:08.127635 (+ 75449us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:08.127672 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:08.127742 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:08.258728 (+130986us) raft_consensus.cc:1671] finished
0806 04:08:08.258747 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:08.261859 (+ 3112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21667,"raft.queue_time_us":321,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":22960,"spinlock_wait_cycles":37992832,"thread_start_us":75002,"threads_started":1,"wal-append.queue_time_us":1352869}
I20260806 04:08:08.262101 8614 raft_consensus.cc:3032] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
W20260806 04:08:08.287178 4876 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 3.073s (SENT)
I20260806 04:08:08.287542 8614 raft_consensus.cc:729] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:08.288067 7932 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:08.288607 4876 leader_election.cc:334] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 3.150s (SENT)
I20260806 04:08:08.289196 4876 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.285145 8282 raft_consensus.cc:3032] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:08.227171 8242 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:08.285056 5857 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 315) took 1615 ms (client timeout 2102 ms). Trace:
I20260806 04:08:08.305817 8527 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:08.307605 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.303618 4813 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:08.318210 4804 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 2.
I20260806 04:08:08.320926 4882 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.289531 4770 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:08.329770 4923 raft_consensus.cc:1203] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Deduplicated request from leader. Original: 2.5->[4.6-6.7] Dedup: 4.6->[6.7-6.7]
I20260806 04:08:08.330359 4923 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260806 04:08:08.330576 8224 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:08.333472 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:08.335430 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.336715 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2018) took 2972 ms. Trace:
I20260806 04:08:08.332545 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:08.351644 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.271100 8120 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.358278 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.303752 8282 raft_consensus.cc:729] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:08.355285 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.360826 4804 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:08.362254 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.362619 8667 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:08.363686 8667 consensus_queue.cc:1225] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Peer 22fe557f852d4415a96fd683f970cc24 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:08:08.353890 4879 leader_election.cc:398] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5.
I20260806 04:08:08.364354 4879 leader_election.cc:401] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:08.364698 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.365622 5867 raft_consensus.cc:2396] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:08.365847 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.366628 5859 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:08.337138 5029 rpcz_store.cc:276] 0806 04:08:05.364615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.567603 (+2202988us) service_pool.cc:227] Handling call
0806 04:08:07.631961 (+ 64358us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:07.968855 (+336894us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:08:07.968885 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:07.968943 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:07.969945 (+ 1002us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:08.336447 (+366502us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b5400016654. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:08.336508 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:08.336658 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10147,"raft.queue_time_us":31149,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":11673,"spinlock_wait_cycles":697936640}
I20260806 04:08:08.365713 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.367488 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.368573 5853 raft_consensus.cc:2379] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:08.358258 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 218559us
I20260806 04:08:08.369755 8527 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 5
W20260806 04:08:08.371739 4992 connection.cc:232] RPC call timeout handler was delayed by -3.1764334689360112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
W20260806 04:08:08.372138 4992 connection.cc:232] RPC call timeout handler was delayed by -3.1579846029635519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
I20260806 04:08:08.372660 5853 raft_consensus.cc:1226] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:08.372771 8164 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.374215 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:08.317919 8619 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:08.322826 8354 raft_consensus.cc:2741] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:08:08.363080 4804 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:08:08.377303 4992 connection.cc:232] RPC call timeout handler was delayed by -3.0753803488332778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24380000000000002s for a response
I20260806 04:08:08.344774 7932 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 04:08:08.382616 4992 connection.cc:232] RPC call timeout handler was delayed by -2.8718886598944664s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260806 04:08:08.383179 4992 connection.cc:232] RPC call timeout handler was delayed by -2.8472753979731351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
W20260806 04:08:08.383613 4992 connection.cc:232] RPC call timeout handler was delayed by -2.756358141079545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
W20260806 04:08:08.383998 4992 connection.cc:232] RPC call timeout handler was delayed by -2.6090929771307856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2739s for a response
W20260806 04:08:08.384337 4992 connection.cc:232] RPC call timeout handler was delayed by -2.4532150849699974s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
W20260806 04:08:08.384838 4992 connection.cc:232] RPC call timeout handler was delayed by -2.389469665940851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
W20260806 04:08:08.385370 4992 connection.cc:232] RPC call timeout handler was delayed by -2.337378767086193s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.42430000000000007s for a response
W20260806 04:08:08.385852 4992 connection.cc:232] RPC call timeout handler was delayed by -2.3305048679467291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
W20260806 04:08:08.386276 4992 connection.cc:232] RPC call timeout handler was delayed by -2.3146007750183344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3007s for a response
W20260806 04:08:08.386715 4992 connection.cc:232] RPC call timeout handler was delayed by -2.155833468772471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2824s for a response
W20260806 04:08:08.387146 4992 connection.cc:232] RPC call timeout handler was delayed by -2.1447625709697604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
W20260806 04:08:08.387689 4992 connection.cc:232] RPC call timeout handler was delayed by -2.0686077948193997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
W20260806 04:08:08.388336 4992 connection.cc:232] RPC call timeout handler was delayed by -1.9476803487632424s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3691s for a response
W20260806 04:08:08.390053 4992 connection.cc:232] RPC call timeout handler was delayed by -1.8970033230725676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3307s for a response
I20260806 04:08:08.390578 4882 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.396749 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.402808 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.358049 8461 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:08.406821 4817 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 9: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
W20260806 04:08:08.415982 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 108037us
W20260806 04:08:08.416182 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 127619us
I20260806 04:08:08.419445 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.419523 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.288609 8614 consensus_queue.cc:260] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.377420 8619 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.407841 5816 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.398540 8463 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:08:08.382087 7932 raft_consensus.cc:686] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:08.428251 5862 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
W20260806 04:08:08.318889 5857 rpcz_store.cc:269] 0806 04:08:06.669859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.672189 (+ 2330us) service_pool.cc:227] Handling call
0806 04:08:08.183821 (+1511632us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:08.284977 (+101156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8666496}
I20260806 04:08:08.433477 4813 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 04:08:08.435022 4876 leader_election.cc:334] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 3.246s (SENT)
I20260806 04:08:08.437650 4876 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.373525 8164 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.412958 8461 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:08.422259 8224 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:08.415472 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:08.451813 4923 raft_consensus.cc:595] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:08.452155 4923 raft_consensus.cc:603] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:08.429247 8619 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:08:08.460963 4926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45408 (request call id 1640) took 1692 ms (client timeout 2134 ms). Trace:
I20260806 04:08:08.458928 4992 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.462618 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.462917 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.418875 8669 consensus_queue.cc:1046] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260806 04:08:08.465261 8281 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:08.465744 8614 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:08.476380 8677 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:08:08.478032 5035 raft_consensus.cc:1226] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 3. Ops: []
I20260806 04:08:08.479712 8644 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:08.479890 8164 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:08.440295 8652 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.450969 8461 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:08.450855 8224 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.472146 5026 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
W20260806 04:08:08.468905 4926 rpcz_store.cc:269] 0806 04:08:06.768812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.487173 (+718361us) service_pool.cc:227] Handling call
0806 04:08:08.442284 (+955111us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54001b7bd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:08.460875 (+ 18591us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1506983680}
I20260806 04:08:08.547933 5037 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
W20260806 04:08:08.551009 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 193657us
W20260806 04:08:08.490641 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 201203us
I20260806 04:08:08.556871 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:08.564458 5034 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:08.575479 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 146412us
I20260806 04:08:08.545919 8641 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:08.578223 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.491844 8520 consensus_queue.cc:1046] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260806 04:08:08.586229 5038 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:08.587931 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.552320 8281 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:08.591693 8520 raft_consensus.cc:3032] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:08:08.592114 8520 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:08.592298 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.593086 5032 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:08.593040 8520 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.594641 8520 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:08:08.601990 8486 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.583s user 0.006s sys 0.001s
W20260806 04:08:08.602873 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 144057us
I20260806 04:08:08.565505 8224 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:08.565722 8401 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:08.619769 5029 raft_consensus.cc:2396] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 04:08:08.610895 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 203160us
I20260806 04:08:08.623257 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:08.623962 5037 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:08.645901 4814 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:08.610745 7932 consensus_queue.cc:237] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.653277 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.654003 5038 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:08.604774 8281 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:08.620936 8401 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.658272 5037 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:08.668344 4992 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:08.666113 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.664300 4814 raft_consensus.cc:2396] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:08.655972 4810 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:08.581503 8534 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:08.672760 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.677114 5035 raft_consensus.cc:1261] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 04:08:08.666723 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 250060us
I20260806 04:08:08.677801 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.678556 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.602630 8486 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.665565 8281 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:08.681095 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:08.560212 8652 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.682377 5030 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:08.683699 5026 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:08.697139 4926 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:08.697800 4926 raft_consensus.cc:595] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:08.655680 8693 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:08.692214 5032 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:08.667752 8565 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.698139 4926 raft_consensus.cc:603] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:08.687951 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:08.674839 8401 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:08.700176 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.700690 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.697355 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:08.706957 4923 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:08.705436 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.691066 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:08.709383 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.710394 4930 raft_consensus.cc:2396] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:08.710906 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.711583 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.712215 8299 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:08.712720 4930 raft_consensus.cc:2379] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:08.713258 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.715478 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.705551 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.716328 4931 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
W20260806 04:08:08.710963 8673 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260806 04:08:08.716656 8565 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.738672 4931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:08.739326 4931 raft_consensus.cc:595] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:08.739682 4931 raft_consensus.cc:603] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:08.740063 4928 raft_consensus.cc:2379] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:08.742408 8447 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:08.743013 8447 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:08.743464 8447 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:08.744180 8486 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:08.744961 8486 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.747543 4809 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:08.742470 4928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:08.751119 4928 raft_consensus.cc:595] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.769060 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:08.769943 5857 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:08.674791 8197 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260806 04:08:08.773448 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 161196us
I20260806 04:08:08.782250 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.786409 5862 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:08.782131 8493 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.787600 5853 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:08.788134 8493 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:08.788893 4818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 383) took 1942 ms (client timeout 1883 ms). Trace:
W20260806 04:08:08.789304 4818 rpcz_store.cc:269] 0806 04:08:06.846114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:06.846248 (+ 134us) service_pool.cc:227] Handling call
0806 04:08:08.788691 (+1942443us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400232154. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:08.788798 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":557590272}
I20260806 04:08:08.789976 8635 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.790464 8635 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:08.701766 8693 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.799561 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:08.811478 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:08.812590 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 130120us
I20260806 04:08:08.832171 4879 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.831977 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:08.840129 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 176489us
W20260806 04:08:08.811781 8640 consensus_meta.cc:287] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.553s user 0.012s sys 0.000s
I20260806 04:08:08.842298 4920 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:08.805464 8693 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:08.766971 8486 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.834371 8565 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:08.854689 4926 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:08.852193 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:08.844807 8391 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s user 0.005s sys 0.014s
W20260806 04:08:08.878029 8158 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.717s user 0.013s sys 0.004s
I20260806 04:08:08.882880 5853 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:08:08.883057 8652 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:08.885404 8242 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:08.885861 8493 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:08.885716 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:08.888799 8527 consensus_meta.cc:287] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.519s user 0.007s sys 0.002s
I20260806 04:08:08.849182 8483 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:08.841521 8644 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.064s
I20260806 04:08:08.889884 8635 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:08.929178 8158 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:08.897660 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 275344us
I20260806 04:08:08.851820 8680 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:08.917296 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.063190 8680 raft_consensus.cc:686] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:09.063944 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.064191 8680 consensus_queue.cc:237] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:09.065244 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.918042 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.071195 4930 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:09.072521 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.073958 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.077219 4813 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:09.077514 8454 raft_consensus.cc:3032] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:09.077987 8454 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:09.078729 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:09.080722 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:09.081655 4810 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:08.918429 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.927218 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:08.888906 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:09.054149 8565 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:08.852679 8640 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.010932 8120 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:08.998452 8619 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.518s user 0.012s sys 0.001s
I20260806 04:08:09.099303 8619 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.099680 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.113196 8486 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:09.070399 8461 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.086772 5857 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
W20260806 04:08:09.164440 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 101788us
I20260806 04:08:08.936041 8527 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5.
I20260806 04:08:09.028354 4923 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:08.936005 4992 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:08.982120 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:08.878381 8709 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260806 04:08:08.927850 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:08.988607 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 315961us
W20260806 04:08:09.012643 8696 log.cc:924] Time spent T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.301s user 0.003s sys 0.000s
W20260806 04:08:09.055923 4768 connection.cc:232] RPC call timeout handler was delayed by -1.1308264969848096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
W20260806 04:08:09.246670 4768 connection.cc:232] RPC call timeout handler was delayed by -1.1182264969684184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260806 04:08:09.247169 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0696264968719333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
W20260806 04:08:09.247604 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0687264970038086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
W20260806 04:08:09.247967 4768 connection.cc:232] RPC call timeout handler was delayed by -1.0390264969319105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
I20260806 04:08:09.249186 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:08.900169 8391 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s user 0.006s sys 0.014s
I20260806 04:08:08.990881 5032 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:08.998219 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.019618 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:09.251240 8461 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:09.251724 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 390) took 1122 ms. Trace:
I20260806 04:08:09.252133 4814 rpcz_store.cc:276] 0806 04:08:08.128996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:08.129142 (+ 146us) service_pool.cc:227] Handling call
0806 04:08:09.251502 (+1122360us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54000af5d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:09.251642 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1097879296}
I20260806 04:08:09.226060 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:09.254343 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 352040us
I20260806 04:08:09.255299 4992 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:09.255653 8224 consensus_meta.cc:287] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.592s user 0.004s sys 0.026s
I20260806 04:08:09.256269 8486 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:09.256069 8224 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:08.889984 5034 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:09.256971 8486 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:09.257378 8486 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:08.983142 8112 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260806 04:08:09.257720 4931 raft_consensus.cc:2379] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:09.257684 8486 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:09.066377 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.259552 8459 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:09.260113 8459 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.261238 4927 raft_consensus.cc:1261] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:08:09.083199 4809 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:09.088883 5028 raft_consensus.cc:2354] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7.
I20260806 04:08:09.087632 4920 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:09.268215 4927 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:09.088045 4930 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:09.270076 4927 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 04:08:09.270720 8578 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:09.083527 8614 consensus_meta.cc:287] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.004s
I20260806 04:08:09.271960 8640 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.272487 8282 consensus_queue.cc:260] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.273831 8282 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:09.235283 8299 consensus_meta.cc:287] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.523s user 0.015s sys 0.004s
I20260806 04:08:09.274479 8299 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:08:09.275477 8632 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.521s user 0.011s sys 0.000s
I20260806 04:08:09.278051 4926 raft_consensus.cc:595] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:09.278487 4926 raft_consensus.cc:603] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:09.097270 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:09.279245 5862 raft_consensus.cc:2379] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:08:09.031553 4770 leader_election.cc:334] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Timed out: RequestConsensusVote RPC to 127.2.248.68:38873 timed out after 1.772s (SENT)
I20260806 04:08:09.284797 4770 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.091212 8242 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260806 04:08:09.226212 5816 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.231284 4919 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
W20260806 04:08:09.303169 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54057.2us
I20260806 04:08:09.233601 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.233788 4995 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:09.230572 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:09.236627 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.266466 4928 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:09.225786 8354 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:09.308697 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:09.307072 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 73359.1us
W20260806 04:08:09.286547 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 255047us
W20260806 04:08:09.310127 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 79624us
W20260806 04:08:09.313966 4770 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873): Timed out: RequestConsensusVote RPC to 127.2.248.68:38873 timed out after 1.936s (SENT)
W20260806 04:08:09.316274 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 152581us
W20260806 04:08:09.291008 8710 log.cc:924] Time spent T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.234s user 0.000s sys 0.007s
I20260806 04:08:09.090530 8565 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.320966 4770 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.311378 8299 raft_consensus.cc:2781] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 04:08:09.306397 8447 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.562s user 0.010s sys 0.009s
I20260806 04:08:09.329324 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.310528 8164 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:09.299721 8614 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:09.341454 4770 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.341864 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:09.343771 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 94401us
I20260806 04:08:09.334513 5816 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:09.346091 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 90870.6us
I20260806 04:08:09.346839 5816 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:09.332089 8709 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:08:09.335459 8617 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:09.332401 8629 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:09.335345 8461 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:08:09.347123 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:09.361236 8664 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:09.341635 8299 raft_consensus.cc:686] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:09.337795 8698 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:09.346683 8520 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:09.363737 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.344458 8578 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:09.366398 4810 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:08:09.374409 4923 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:09.374442 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2216) took 1112 ms. Trace:
I20260806 04:08:09.333621 8354 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.342110 8164 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:09.383219 4817 rpcz_store.cc:276] 0806 04:08:08.262741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:08.464210 (+201469us) service_pool.cc:227] Handling call
0806 04:08:08.464576 (+ 366us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:08.661374 (+196798us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:08.661406 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:08.661874 (+ 468us) log.cc:842] Serialized 26 byte log entry
0806 04:08:08.926304 (+264430us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:08.926354 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:08.926423 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:09.374231 (+447808us) raft_consensus.cc:1671] finished
0806 04:08:09.374251 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:09.374385 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26420,"raft.queue_time_us":125190,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":19006,"thread_start_us":427431,"threads_started":2,"wal-append.queue_time_us":649909}
I20260806 04:08:09.368221 8629 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:09.358453 8709 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:09.387588 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 48115us
I20260806 04:08:09.375530 4813 raft_consensus.cc:1261] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:08:09.372948 8158 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.364123 8614 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:09.384202 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.368359 8635 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:09.342381 8667 consensus_queue.cc:1046] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260806 04:08:09.341075 8447 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.412565 4992 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.376477 8664 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:09.419345 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.400262 8578 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:08:09.421448 8578 raft_consensus.cc:686] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:09.422045 8691 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:09.422873 8691 consensus_queue.cc:1035] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:09.425621 8614 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:08:09.427204 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.429023 8698 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:09.429489 8698 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:09.429929 8698 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:09.430864 5812 leader_election.cc:334] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.030s (SENT)
I20260806 04:08:09.431417 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.432742 8691 consensus_queue.cc:1046] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:08:09.386683 8461 raft_consensus.cc:686] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:09.394058 8164 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:09.375221 8617 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.409919 8709 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:09.412283 8640 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:09.445080 8622 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 1.565s user 0.011s sys 0.006s
I20260806 04:08:09.447639 8493 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 04:08:09.447780 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 64446.5us
W20260806 04:08:09.448396 5860 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 318) took 1788 ms (client timeout 2299 ms). Trace:
W20260806 04:08:09.448598 8716 log.cc:924] Time spent T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
I20260806 04:08:09.398737 8629 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:09.412326 8644 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:09.476964 5860 rpcz_store.cc:269] 0806 04:08:07.659748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.659891 (+ 143us) service_pool.cc:227] Handling call
0806 04:08:09.448194 (+1788303us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b540051ed54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:09.448343 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1563019008}
W20260806 04:08:09.442811 8463 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.004s user 0.022s sys 0.007s
I20260806 04:08:09.456769 5859 raft_consensus.cc:1226] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:09.497859 4992 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.414105 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:09.501885 8730 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:08:09.502830 8401 consensus_meta.cc:287] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.762s user 0.016s sys 0.000s
I20260806 04:08:09.503337 8401 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:09.437150 8718 log.cc:924] Time spent T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260806 04:08:09.504719 8708 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:09.505594 8724 consensus_queue.cc:1035] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260806 04:08:09.435252 8664 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:09.484572 8617 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.484977 5854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 321) took 1516 ms. Trace:
W20260806 04:08:09.485868 8693 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.589s user 0.007s sys 0.006s
W20260806 04:08:09.495160 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 148411us
I20260806 04:08:09.481810 4879 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.495846 4924 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:09.461545 8565 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:09.521567 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 130944us
W20260806 04:08:09.517158 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 78059.3us
W20260806 04:08:09.513687 8281 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.800s user 0.013s sys 0.006s
I20260806 04:08:09.505613 8401 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:09.509332 8619 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.537328 4931 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260806 04:08:09.532317 8354 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.533283 8722 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260806 04:08:09.474181 8640 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.534745 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:09.481204 8641 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260806 04:08:09.545343 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:09.542824 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 50445.9us
I20260806 04:08:09.546198 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.544395 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:09.496608 8644 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:09.545343 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:09.549638 5860 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:09.550853 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.551980 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.472812 8299 consensus_queue.cc:237] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.553510 5034 raft_consensus.cc:2379] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:09.553699 4817 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:09.555107 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:09.555421 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.555591 4770 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.556190 4816 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:09.555781 5038 raft_consensus.cc:2362] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
I20260806 04:08:09.559157 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:09.559844 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:09.561750 5037 raft_consensus.cc:2396] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:08:09.556005 5028 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
W20260806 04:08:09.563474 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 102855us
I20260806 04:08:09.560397 5863 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:09.564242 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.565320 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.566294 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.499298 8578 consensus_queue.cc:237] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:09.566015 5038 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:09.572070 4992 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.568221 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.549939 8737 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:09.569051 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.570047 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:09.575906 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:09.576259 5026 raft_consensus.cc:2362] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4. Re-sending same reply.
I20260806 04:08:09.576720 8619 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:09.577123 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.577246 8619 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:09.577834 8680 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:08:09.570487 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 119700us
I20260806 04:08:09.571017 4770 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.578387 8619 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:09.579744 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.568439 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:09.582068 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.583606 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.519759 8693 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.573266 5030 raft_consensus.cc:2396] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260806 04:08:09.575711 5029 raft_consensus.cc:2362] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
I20260806 04:08:09.586930 5035 raft_consensus.cc:2354] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6.
I20260806 04:08:09.587785 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:09.584259 4925 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:09.538744 8281 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.588618 4810 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:09.597458 4996 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.615797 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:09.516322 5854 rpcz_store.cc:276] 0806 04:08:07.985724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:07.992034 (+ 6310us) service_pool.cc:227] Handling call
0806 04:08:08.020648 (+ 28614us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:08.156801 (+136153us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:09.484666 (+1327865us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x7b54004fced4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:09.484884 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110187392}
I20260806 04:08:09.569121 8617 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:09.628260 4808 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:09.624845 8730 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260806 04:08:09.633265 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:09.633329 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:09.538058 8461 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:09.643136 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 103251us
I20260806 04:08:09.640800 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:09.582264 8447 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:09.647148 8735 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260806 04:08:09.647687 5857 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:09.648378 8302 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:09.618460 8565 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:09.654551 4816 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:09.572224 8354 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:09.655205 4818 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:09.660779 8741 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:09.664583 8299 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:08:09.666896 5866 raft_consensus.cc:2396] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260806 04:08:09.667639 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 88094.7us
I20260806 04:08:09.655905 8722 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:09.633759 8401 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:09.634410 8617 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:09.704697 8617 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:09.704768 5856 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:09.705915 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 597) took 1542 ms. Trace:
I20260806 04:08:09.706348 5856 rpcz_store.cc:276] 0806 04:08:08.163457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:08.167497 (+ 4040us) service_pool.cc:227] Handling call
0806 04:08:09.704181 (+1536684us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:09.704489 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:09.705499 (+ 1010us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:09.705837 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6233984}
I20260806 04:08:09.560825 8732 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.103s
I20260806 04:08:09.710691 8405 raft_consensus.cc:3032] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:09.711189 8405 raft_consensus.cc:729] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:09.712225 8405 consensus_queue.cc:260] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.704617 8401 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:09.718029 4925 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:08:09.677081 8565 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:09.700691 8354 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:09.738929 5034 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:08:09.713902 8493 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:09.735948 8401 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:09.738667 8741 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:09.743427 8565 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:09.679641 8629 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.616951 8737 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:09.735948 8708 consensus_queue.cc:1046] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260806 04:08:09.993844 4770 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:09.811007 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:09.928254 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 362742us
I20260806 04:08:09.871791 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:09.993361 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 319551us
I20260806 04:08:09.997565 4817 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:09.798488 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:09.899442 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:09.776712 8741 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.999146 5863 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:10.001142 8741 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:10.002496 4813 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:09.998066 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:09.873497 5812 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.560s (SENT)
W20260806 04:08:10.004453 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 339028us
I20260806 04:08:09.747694 8354 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:09.995523 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 258816us
I20260806 04:08:10.005432 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:10.009369 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:09.806197 8299 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:10.029250 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:10.031188 8282 consensus_meta.cc:287] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.757s user 0.014s sys 0.000s
W20260806 04:08:09.900813 8733 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
W20260806 04:08:09.995952 8749 log.cc:924] Time spent T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.249s user 0.001s sys 0.000s
W20260806 04:08:09.901513 8763 log.cc:924] Time spent T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260806 04:08:10.003603 5036 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:09.908254 5902 heartbeater.cc:644] Failed to heartbeat to 127.2.248.126:44289 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.2.248.126:44289 timed out after 15.000s (SENT)
I20260806 04:08:10.021243 4992 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:09.898186 5854 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:09.981671 8405 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:10.059080 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 221485us
I20260806 04:08:10.080247 4816 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
W20260806 04:08:09.899798 8486 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.641s user 0.013s sys 0.000s
W20260806 04:08:10.004383 8164 consensus_meta.cc:287] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.544s user 0.008s sys 0.003s
I20260806 04:08:10.081691 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:09.804092 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:10.093950 4882 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:10.083745 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 44396.4us
I20260806 04:08:10.085619 4804 raft_consensus.cc:1261] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260806 04:08:10.098017 4996 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:09.802438 8281 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.807796 8693 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:09.831717 8640 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:09.806903 8644 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:09.780522 8302 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:10.103982 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 230534us
W20260806 04:08:10.005913 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 997e4d9ff3ca465199a888e8af23ed7b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
W20260806 04:08:10.127065 4651 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 121199us
I20260806 04:08:10.129350 8693 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:10.107939 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:10.130247 8693 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:09.914212 8566 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:10.133328 8566 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:10.109903 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:10.134344 4816 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:10.049928 8299 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.138680 8299 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:10.140192 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:10.140431 5812 leader_election.cc:334] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.741s (SENT)
I20260806 04:08:10.140847 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:10.143378 8532 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:10.151000 8693 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:09.832396 8680 consensus_queue.cc:1035] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.267s
I20260806 04:08:10.062414 8667 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:09.767089 8493 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:10.189332 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 105875us
I20260806 04:08:10.108098 5856 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:10.174642 8302 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.199858 8644 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:10.200278 8741 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:10.099661 8486 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.189416 8775 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:10.201014 8566 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 04:08:10.223548 8617 consensus_meta.cc:287] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.004s
I20260806 04:08:10.099846 8614 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:10.202632 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:10.194020 4879 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:10.007555 7714 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.114s
W20260806 04:08:10.222525 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 102598us
W20260806 04:08:10.195712 8520 consensus_meta.cc:287] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.808s user 0.010s sys 0.007s
I20260806 04:08:10.239892 5860 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 04:08:10.202426 8737 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:10.099975 8164 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.267392 8566 raft_consensus.cc:686] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:10.113417 8619 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:10.206863 8749 log.cc:924] Time spent T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.113s user 0.001s sys 0.007s
I20260806 04:08:10.263621 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:10.228827 8644 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:10.234368 8453 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:10.231572 8741 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:10.310770 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 116854us
W20260806 04:08:10.367355 5812 leader_election.cc:334] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.883s (SENT)
W20260806 04:08:10.430586 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 63285.7us
I20260806 04:08:10.382259 8629 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:10.370620 8486 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:10.437672 8745 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:10.235443 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 214341us
I20260806 04:08:10.438269 8745 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.439792 4992 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:10.440048 8401 consensus_meta.cc:287] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.658s user 0.010s sys 0.000s
I20260806 04:08:10.440543 8401 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.441203 4992 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.306399 8520 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:08:10.371937 8664 consensus_meta.cc:287] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.849s user 0.010s sys 0.004s
W20260806 04:08:10.381258 8698 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.951s user 0.007s sys 0.015s
W20260806 04:08:10.384821 8709 consensus_meta.cc:287] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.904s user 0.010s sys 0.005s
I20260806 04:08:10.409848 4928 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:10.415655 5862 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 04:08:10.418527 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 49484.2us
I20260806 04:08:10.380347 8641 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:10.266592 8617 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.360904 5860 pending_rounds.cc:76] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Aborting all ops after (but not including) 6
W20260806 04:08:10.432070 5812 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 2.270s (SENT) [suppressed 39 similar messages]
I20260806 04:08:10.306151 8614 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.435079 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:10.436761 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:10.419703 8644 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:10.431881 8741 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:08:10.460419 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 59739.8us
I20260806 04:08:10.467564 5860 pending_rounds.cc:97] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 04:08:10.493749 5860 raft_consensus.cc:2866] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:10.468201 8401 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:10.471936 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59506 (request call id 32) took 1630 ms. Trace:
I20260806 04:08:10.473097 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2234) took 1099 ms. Trace:
I20260806 04:08:10.412638 8619 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.467120 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:10.266700 8775 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.489539 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:10.440605 8745 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:10.495779 4805 rpcz_store.cc:276] 0806 04:08:09.374163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.374298 (+ 135us) service_pool.cc:227] Handling call
0806 04:08:09.374597 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:09.383449 (+ 8852us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.383481 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:09.383515 (+ 34us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:09.383541 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:10.472802 (+1089261us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b54002a44d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:10.472863 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:10.473024 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19934464}
I20260806 04:08:10.478755 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:10.505157 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:10.466549 8486 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:10.506915 4817 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:10.495038 4926 rpcz_store.cc:276] 0806 04:08:08.841699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:08.841938 (+ 239us) service_pool.cc:227] Handling call
0806 04:08:10.471626 (+1629688us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54003d1154. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9954bc3 00007f23de89e81a 00007f23d7f91d04 00007f23d7f91c92 00007f23d7f91bc1 00007f23d7f91b3c 00007f23d7f90df2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:10.471823 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879091072}
I20260806 04:08:10.507157 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:10.508436 4816 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:10.462806 8641 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:10.500386 8282 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:10.501847 8792 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260806 04:08:10.501827 8302 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:10.510304 8772 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260806 04:08:10.510857 4876 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.511412 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260806 04:08:10.501760 8759 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
I20260806 04:08:10.449440 8242 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.047s
I20260806 04:08:10.483474 8708 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:10.454018 8664 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.514969 4928 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:10.530000 4768 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:10.531100 4805 raft_consensus.cc:1261] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:08:10.515620 5867 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:10.518801 8282 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:10.519497 8776 log.cc:924] Time spent T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
W20260806 04:08:10.514880 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 82842us
I20260806 04:08:10.453641 8698 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.533111 5867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 330) took 1192 ms. Trace:
I20260806 04:08:10.523772 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:10.521347 8641 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:10.466409 8709 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.538354 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.536407 5867 rpcz_store.cc:276] 0806 04:08:09.341078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.341252 (+ 174us) service_pool.cc:227] Handling call
0806 04:08:10.478509 (+1137257us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:10.533051 (+ 54542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13158784}
I20260806 04:08:10.540690 4768 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:10.541261 4805 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:10.528082 8164 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:10.545069 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:10.545964 4928 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:10.551623 4924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 393) took 2205 ms (client timeout 2270 ms). Trace:
W20260806 04:08:10.552099 4924 rpcz_store.cc:269] 0806 04:08:08.345836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:08.346240 (+ 404us) service_pool.cc:227] Handling call
0806 04:08:10.551434 (+2205194us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b54000f6754. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:10.551557 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1970334464}
W20260806 04:08:10.558900 8760 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.317s user 0.001s sys 0.000s
I20260806 04:08:10.554833 8708 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:08:10.567624 5036 consensus_meta.cc:287] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.503s user 0.012s sys 0.000s
I20260806 04:08:10.574385 5036 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:10.573369 4992 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:10.574934 5036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2023) took 2104 ms (client timeout 2765 ms). Trace:
W20260806 04:08:10.575378 5036 rpcz_store.cc:269] 0806 04:08:08.470971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.735710 (+1264739us) service_pool.cc:227] Handling call
0806 04:08:10.574875 (+839165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:10.516877 8566 consensus_queue.cc:237] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.568562 5853 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 04:08:10.547796 8745 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:10.582510 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1670) took 1524 ms. Trace:
I20260806 04:08:10.622869 4930 rpcz_store.cc:276] 0806 04:08:09.057936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.270852 (+212916us) service_pool.cc:227] Handling call
0806 04:08:09.271177 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:09.271372 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.271395 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:09.721745 (+450350us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b2c0000f228. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d833de11 00007f23d834d442 00007f23d9995c11 00007f23d995a58a 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772
0806 04:08:09.722926 (+ 1181us) log.cc:842] Serialized 26 byte log entry
0806 04:08:09.996616 (+273690us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:09.996652 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:09.996724 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:10.578052 (+581328us) raft_consensus.cc:1671] finished
0806 04:08:10.578072 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:10.582437 (+ 4365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39064064,"thread_start_us":227652,"threads_started":1,"wal-append.queue_time_us":492630}
I20260806 04:08:10.591528 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 399) took 1041 ms. Trace:
W20260806 04:08:10.599852 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 77227us
I20260806 04:08:10.625280 4804 rpcz_store.cc:276] 0806 04:08:09.550368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.550503 (+ 135us) service_pool.cc:227] Handling call
0806 04:08:09.558820 (+ 8317us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:10.097628 (+538808us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:10.591183 (+493555us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b5400231c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:10.591425 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1210392832}
W20260806 04:08:10.573141 8774 log.cc:924] Time spent T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260806 04:08:10.591361 4816 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:10.629069 4996 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:10.593267 8354 consensus_meta.cc:287] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.512s user 0.010s sys 0.001s
I20260806 04:08:10.647604 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:10.653507 4930 raft_consensus.cc:2362] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5. Re-sending same reply.
I20260806 04:08:10.655110 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:10.655478 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:10.656226 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.583315 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 605) took 1203 ms. Trace:
I20260806 04:08:10.669639 4995 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:10.604722 8784 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
W20260806 04:08:10.674917 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 111498us
I20260806 04:08:10.683475 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:10.684194 4924 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:10.656370 8578 raft_consensus.cc:2781] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:10.641263 8801 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:10.658214 8302 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:10.691838 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:10.691241 4928 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
W20260806 04:08:10.699781 5812 leader_election.cc:334] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.642s (SENT)
I20260806 04:08:10.706861 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:10.579126 8789 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.166s
I20260806 04:08:10.712556 4924 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:10.710354 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:10.679013 8532 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:10.679288 8401 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:10.646770 8164 consensus_queue.cc:1035] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260806 04:08:10.657866 8629 consensus_queue.cc:1035] T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260806 04:08:10.723196 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:10.655061 8354 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.719340 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:10.723942 4805 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:10.731743 4995 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.734093 4816 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:10.709635 8617 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:10.736330 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 58438.2us
I20260806 04:08:10.718761 8732 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:10.717443 8779 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:10.714217 8578 raft_consensus.cc:686] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:10.669138 5857 rpcz_store.cc:276] 0806 04:08:09.379732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.382229 (+ 2497us) service_pool.cc:227] Handling call
0806 04:08:10.583024 (+1200795us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54004448d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:10.583240 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316244352}
I20260806 04:08:10.754743 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1675) took 1175 ms. Trace:
I20260806 04:08:10.732626 8532 raft_consensus.cc:686] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:10.732595 8614 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:10.757083 4923 rpcz_store.cc:276] 0806 04:08:09.579330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.580887 (+ 1557us) service_pool.cc:227] Handling call
0806 04:08:09.581234 (+ 347us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:09.827785 (+246551us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.827831 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:10.005081 (+177250us) log.cc:842] Serialized 26 byte log entry
0806 04:08:10.233134 (+228053us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:10.233173 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:10.233235 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:10.754503 (+521268us) raft_consensus.cc:1671] finished
0806 04:08:10.754521 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:10.754679 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10531,"raft.queue_time_us":128225,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":337222,"threads_started":2,"wal-append.queue_time_us":374576}
I20260806 04:08:10.770401 4992 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.745229 8242 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:10.758410 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:10.777029 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 266256us
I20260806 04:08:10.761080 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:10.779433 5867 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:10.734531 8698 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:10.779812 4876 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.741127 8401 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:10.758322 4995 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.781569 4770 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:10.782949 8698 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:10.783349 8698 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:10.783733 8698 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:10.784590 4816 raft_consensus.cc:2454] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:10.785774 5816 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:10.748523 8282 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:10.748370 8779 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:10.745651 8664 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:10.748054 8775 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:10.720372 8302 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.709128 8785 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.232s
I20260806 04:08:10.753281 8782 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:10.792879 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 90264.3us
I20260806 04:08:10.806149 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:10.819577 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:10.820326 4926 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:10.790418 8708 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:10.790184 8709 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:10.810163 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 53707.2us
W20260806 04:08:10.810948 8794 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260806 04:08:10.813082 8726 raft_consensus.cc:979] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:08:10.819221 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:10.819166 5867 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:08:10.790504 8486 raft_consensus.cc:2741] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:08:10.790829 8619 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:10.832976 4768 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:10.787680 8732 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:10.854477 8619 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:10.855221 8619 raft_consensus.cc:686] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:10.854704 8732 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.856253 8619 consensus_queue.cc:237] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.793949 8779 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:10.863518 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1666) took 1813 ms. Trace:
I20260806 04:08:10.863981 4920 rpcz_store.cc:276] 0806 04:08:09.050530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.268861 (+218331us) service_pool.cc:227] Handling call
0806 04:08:09.269189 (+ 328us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:09.269419 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.269447 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:10.589164 (+1319717us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b2c0000f228. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d833de11 00007f23d834d442 00007f23d9995c11 00007f23d995a58a 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772
0806 04:08:10.589617 (+ 453us) log.cc:842] Serialized 26 byte log entry
0806 04:08:10.739443 (+149826us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:10.739484 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:10.739573 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:10.863200 (+123627us) raft_consensus.cc:1671] finished
0806 04:08:10.863218 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:10.863447 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43255680,"thread_start_us":149399,"threads_started":1,"wal-append.queue_time_us":238240}
I20260806 04:08:10.866129 4770 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:10.866546 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:10.722944 8801 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.878166 4920 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:08:10.792085 8814 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.148s
I20260806 04:08:10.880793 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:10.883934 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 130103us
I20260806 04:08:10.890772 4876 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:10.875991 8282 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:08:10.909348 5812 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.109s (SENT)
I20260806 04:08:10.914705 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:10.923542 5867 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
W20260806 04:08:10.960454 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 74188.5us
I20260806 04:08:10.961462 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:10.965505 8617 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:10.968040 4876 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:10.901873 8401 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:10.908001 8486 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:10.970752 4992 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:10.901822 8614 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:10.976837 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:10.913113 8282 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:10.973353 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:10.876212 8578 consensus_queue.cc:237] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:10.958833 8629 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:10.972987 8772 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 04:08:11.025559 8785 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:08:10.977941 8755 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260806 04:08:11.025759 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 55126.4us
I20260806 04:08:10.996913 8354 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.033134 8282 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:08:11.044225 8796 log.cc:924] Time spent T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.252s user 0.000s sys 0.001s
I20260806 04:08:10.971046 4923 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:11.028185 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:11.018873 8565 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.006s user 0.013s sys 0.001s
I20260806 04:08:10.968040 8302 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:10.925911 8532 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:10.997741 8401 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.027854 8617 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.055027 8808 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:11.075155 8822 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.161s
I20260806 04:08:11.075978 8565 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.040562 8698 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.049288 8824 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:11.043076 8812 raft_consensus.cc:2781] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 04:08:11.079394 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 102568us
W20260806 04:08:11.111981 8795 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260806 04:08:11.113648 4876 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.049361 8811 raft_consensus.cc:1067] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
W20260806 04:08:11.075672 8832 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260806 04:08:11.111506 8838 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.012s
I20260806 04:08:11.080830 4992 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.039182 8614 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.137105 8565 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:11.137921 5856 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.899s user 0.009s sys 0.001s
I20260806 04:08:11.138110 4809 raft_consensus.cc:2379] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:11.139302 8812 raft_consensus.cc:686] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:11.142833 8698 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.143553 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:11.102725 8808 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:11.145210 8820 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:11.145998 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:11.183277 5862 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:11.149192 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.185629 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:11.149927 8824 consensus_queue.cc:1035] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:08:11.151257 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2226) took 2348 ms. Trace:
I20260806 04:08:11.152768 8617 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:11.155082 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2049) took 1604 ms (client timeout 1810 ms). Trace:
I20260806 04:08:11.159619 4923 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:11.158921 4805 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:11.186586 4818 rpcz_store.cc:276] 0806 04:08:08.802589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:08.802981 (+ 392us) service_pool.cc:227] Handling call
0806 04:08:08.803340 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:09.702264 (+898924us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:11.151003 (+1448739us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b54002710d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:11.151190 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1951890176}
W20260806 04:08:11.159982 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 257694us
W20260806 04:08:11.187212 5034 rpcz_store.cc:269] 0806 04:08:09.550891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.565056 (+ 14165us) service_pool.cc:227] Handling call
0806 04:08:11.154838 (+1589782us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000bde54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:11.155002 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":241366784}
I20260806 04:08:11.169903 5867 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:11.180487 8810 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 04:08:11.191284 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:11.192405 4814 raft_consensus.cc:2354] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7.
I20260806 04:08:11.149950 8836 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:11.243327 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 162859us
I20260806 04:08:11.244987 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:11.186933 8808 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:11.173054 8782 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:11.247064 5812 leader_election.cc:334] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.206s (SENT)
W20260806 04:08:11.253734 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 68207.7us
I20260806 04:08:11.254626 4876 leader_election.cc:398] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7.
I20260806 04:08:11.254964 4876 leader_election.cc:401] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260806 04:08:11.194025 8644 consensus_meta.cc:287] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.691s user 0.011s sys 0.000s
I20260806 04:08:11.170130 5856 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:08:11.182883 8820 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:11.188310 8801 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.191781 8180 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:08:11.270582 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 345688us
I20260806 04:08:11.243680 8836 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:11.293993 4876 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:11.283625 5856 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 676) took 1464 ms (client timeout 1543 ms). Trace:
I20260806 04:08:11.270087 8820 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:11.292994 8801 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:11.297040 8224 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.321714 8827 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
I20260806 04:08:11.263439 8644 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.324945 8819 raft_consensus.cc:2741] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260806 04:08:11.297959 8811 raft_consensus.cc:3037] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:11.294871 8782 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:08:11.296243 8836 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:11.314711 5854 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: [4.6-7.7]
W20260806 04:08:11.351738 4926 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.522s user 0.011s sys 0.001s
I20260806 04:08:11.353663 4923 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:08:11.363953 4926 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:08:11.326372 8401 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:11.303512 5856 rpcz_store.cc:269] 0806 04:08:09.803478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.803710 (+ 232us) service_pool.cc:227] Handling call
0806 04:08:11.267954 (+1464244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:11.367897 8827 raft_consensus.cc:3032] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 04:08:11.388688 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 144571us
I20260806 04:08:11.387183 8827 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:11.394026 8830 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260806 04:08:11.394387 8827 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.395944 8827 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:11.396476 8644 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.318624 8812 consensus_queue.cc:237] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.398821 8686 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:11.399461 8686 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.402123 8454 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.403379 8454 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:08:11.405999 8741 consensus_meta.cc:287] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.897s user 0.007s sys 0.004s
I20260806 04:08:11.381546 8833 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:11.374673 8782 raft_consensus.cc:686] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:11.418828 5862 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
W20260806 04:08:11.427291 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 273534us
I20260806 04:08:11.401731 8686 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:08:11.447317 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:11.448453 4816 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:11.452245 5816 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:11.460886 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 215993us
I20260806 04:08:11.408442 8631 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:11.466117 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:11.478437 4770 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:11.406778 8803 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:11.363945 8801 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.406495 8741 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.519338 8617 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:11.464779 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:11.523185 8732 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260806 04:08:11.524546 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 84027.8us
I20260806 04:08:11.574321 4992 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:11.576088 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 282178us
I20260806 04:08:11.580271 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:11.577266 5867 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:11.581265 4805 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:11.585799 4770 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:11.578378 5856 raft_consensus.cc:2362] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8. Re-sending same reply.
I20260806 04:08:11.571540 8782 consensus_queue.cc:237] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:11.591656 8840 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260806 04:08:11.592324 8493 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:11.587224 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:11.596439 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:11.600224 4770 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:11.601517 8822 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:11.602021 8822 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:08:11.602201 8779 consensus_meta.cc:287] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.724s user 0.008s sys 0.000s
I20260806 04:08:11.602748 8779 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.604184 8808 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.605477 4928 raft_consensus.cc:1226] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:11.575423 8732 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:11.608728 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:11.612427 4992 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:11.606976 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1684) took 1201 ms. Trace:
I20260806 04:08:11.613598 4925 rpcz_store.cc:276] 0806 04:08:10.405645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:10.586728 (+181083us) service_pool.cc:227] Handling call
0806 04:08:10.655652 (+ 68924us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:10.923275 (+267623us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:08:11.178579 (+255304us) log.cc:842] Serialized 14 byte log entry
0806 04:08:11.296115 (+117536us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:11.296600 (+ 485us) log.cc:842] Serialized 26 byte log entry
0806 04:08:11.296757 (+ 157us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:11.296791 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:11.296915 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:11.606735 (+309820us) raft_consensus.cc:1671] finished
0806 04:08:11.606756 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:11.606917 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42438,"raft.queue_time_us":141665,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":16318,"thread_start_us":535108,"threads_started":3,"wal-append.queue_time_us":272300}
I20260806 04:08:11.577600 8617 raft_consensus.cc:686] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:11.623519 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.630262 4876 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:11.638998 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 51865.9us
W20260806 04:08:11.640070 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 997e4d9ff3ca465199a888e8af23ed7b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:11.638062 8644 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:11.643605 8644 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:11.644176 8644 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:11.644556 4876 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.644505 8617 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.622848 8281 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:11.622300 8614 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.639444 8833 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:11.643537 8862 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:11.605180 8779 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.646610 4876 leader_election.cc:398] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6.
I20260806 04:08:11.612128 8732 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:11.661836 8224 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:11.666074 8224 raft_consensus.cc:686] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:11.667328 8644 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.669651 8644 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.649325 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:11.671382 5856 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:11.675770 4876 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.667214 8224 consensus_queue.cc:237] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.683463 8401 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:11.647029 8846 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:11.684480 8708 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:11.649772 8862 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.632238 8808 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.690917 5854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 336) took 1015 ms. Trace:
I20260806 04:08:11.649152 8833 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:11.647351 8281 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:11.491067 8803 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.711824 8744 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:11.691089 8578 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:11.702934 8708 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:11.743057 8708 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:11.740314 8281 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.712661 5854 rpcz_store.cc:276] 0806 04:08:10.675064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:10.675212 (+ 148us) service_pool.cc:227] Handling call
0806 04:08:11.269043 (+593831us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:11.373261 (+104218us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:11.685151 (+311890us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54004fced0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:11.690832 (+ 5681us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1196708224}
I20260806 04:08:11.719501 8833 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:11.719918 8808 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:11.750396 8744 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:11.773474 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 2242) took 1975 ms (client timeout 2545 ms). Trace:
W20260806 04:08:11.774037 4809 rpcz_store.cc:269] 0806 04:08:09.797878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.798094 (+ 216us) service_pool.cc:227] Handling call
0806 04:08:11.773231 (+1975137us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400271fd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:11.773399 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2586063232}
I20260806 04:08:11.767017 8808 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.784907 8741 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.676455 8822 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.789479 8756 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:11.714349 8836 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:11.780320 8808 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.790292 8756 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:11.781462 8493 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:11.801563 8744 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.820331 8801 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.821995 8589 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:11.823940 8734 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 2.327s user 0.004s sys 0.013s
I20260806 04:08:11.822490 8756 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:11.825059 8493 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.779279 8820 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.829833 8862 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:11.835748 8589 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.836447 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1385) took 1828 ms. Trace:
I20260806 04:08:11.840188 4817 rpcz_store.cc:276] 0806 04:08:10.007983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:10.042387 (+ 34404us) service_pool.cc:227] Handling call
0806 04:08:10.096949 (+ 54562us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:10.507347 (+410398us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:11.836169 (+1328822us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b5400380254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:11.836370 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":910794112}
I20260806 04:08:11.836723 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 614) took 1018 ms. Trace:
W20260806 04:08:11.842550 8828 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
W20260806 04:08:11.835103 8282 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.782s user 0.009s sys 0.006s
W20260806 04:08:11.848389 8641 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.302s user 0.016s sys 0.000s
I20260806 04:08:11.892163 8820 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.845089 5857 rpcz_store.cc:276] 0806 04:08:10.818676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:10.829457 (+ 10781us) service_pool.cc:227] Handling call
0806 04:08:10.879859 (+ 50402us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:08:10.920533 (+ 40674us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:10.959098 (+ 38565us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 04:08:10.960109 (+ 1011us) log.cc:842] Serialized 78 byte log entry
0806 04:08:11.045171 (+ 85062us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:11.107012 (+ 61841us) log.cc:842] Serialized 14 byte log entry
0806 04:08:11.169308 (+ 62296us) log.cc:842] Serialized 14 byte log entry
0806 04:08:11.211522 (+ 42214us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:11.243122 (+ 31600us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:11.836225 (+593103us) raft_consensus.cc:1671] finished
0806 04:08:11.836277 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:11.836582 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":66188,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":8670,"raft.queue_time_us":30710,"raft.run_cpu_time_us":547,"raft.run_wall_time_us":80652,"thread_start_us":73461,"threads_started":1,"wal-append.queue_time_us":764549}
I20260806 04:08:11.873543 8282 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.879179 8836 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.928110 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1672) took 2674 ms. Trace:
I20260806 04:08:11.929332 8282 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:11.928771 4919 rpcz_store.cc:276] 0806 04:08:09.253775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.308869 (+ 55094us) service_pool.cc:227] Handling call
0806 04:08:09.331911 (+ 23042us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:09.332205 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.332241 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:09.338634 (+ 6393us) log.cc:842] Serialized 26 byte log entry
0806 04:08:09.456358 (+117724us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:09.456395 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:09.456472 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:11.806438 (+2349966us) spinlock_profiling.cc:243] Waited 8.38 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:11.927681 (+121243us) raft_consensus.cc:1671] finished
0806 04:08:11.927712 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:11.928019 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19274624,"thread_start_us":89211,"threads_started":1,"wal-append.queue_time_us":142754}
I20260806 04:08:11.934840 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:11.935946 5857 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:11.938835 8391 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:11.939684 8391 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.941818 5862 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:11.955694 8493 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:11.957230 5853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 331) took 2578 ms. Trace:
I20260806 04:08:11.882875 8641 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.974077 5857 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:08:11.957963 5853 rpcz_store.cc:276] 0806 04:08:09.378701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.381630 (+ 2929us) service_pool.cc:227] Handling call
0806 04:08:09.413987 (+ 32357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:10.621232 (+1207245us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:11.956665 (+1335433us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540059ecd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:11.957057 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2590725376}
I20260806 04:08:11.977818 8459 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:11.981791 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 369462us
I20260806 04:08:12.002997 4992 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:11.992014 4770 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:12.004578 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:12.005064 8779 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:08:12.006309 8779 raft_consensus.cc:686] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:12.008843 8779 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:11.963533 8732 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:12.000867 8827 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.604s user 0.010s sys 0.006s
I20260806 04:08:12.020318 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:12.016889 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1694) took 1185 ms. Trace:
I20260806 04:08:12.021198 4809 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:12.017396 8726 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:12.019871 8732 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865)
I20260806 04:08:12.000682 8641 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:12.005277 8302 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:08:12.028545 8811 consensus_meta.cc:287] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.677s user 0.014s sys 0.003s
I20260806 04:08:12.029254 8811 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7.
I20260806 04:08:12.021293 4930 rpcz_store.cc:276] 0806 04:08:10.831085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:10.832021 (+ 936us) service_pool.cc:227] Handling call
0806 04:08:10.832413 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:10.832634 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:10.880974 (+ 48340us) log.cc:842] Serialized 14 byte log entry
0806 04:08:11.972192 (+1091218us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b18000f8c80. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23dd802bcf 00007f23d6a327b6 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d
0806 04:08:12.016239 (+ 44047us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b18000f8c80. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a2f155 00007f23d99523d0 00007f23d9964576 00007f23d99750d7
0806 04:08:12.016447 (+ 208us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:12.016525 (+ 78us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:12.016587 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:12.016691 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:12.016831 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47583,"prepare.queue_time_us":68499,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":2207,"raft.queue_time_us":90924,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":18859,"spinlock_wait_cycles":72572032,"thread_start_us":122485,"threads_started":1}
W20260806 04:08:12.054243 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 476762us
I20260806 04:08:12.052809 4809 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:12.056656 5816 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:12.056808 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:12.060935 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 420771us
I20260806 04:08:12.063482 4809 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:11.996670 8708 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:12.068841 4879 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.003s sys 0.020s
W20260806 04:08:12.069257 4879 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.003s sys 0.020s
I20260806 04:08:12.066778 8667 raft_consensus.cc:2781] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:12.005059 4992 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:12.080649 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 75419us
I20260806 04:08:12.079037 8302 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:12.095204 4809 raft_consensus.cc:1226] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:12.079504 8667 raft_consensus.cc:686] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:12.123473 4992 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.147764 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:12.148753 4930 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:12.146818 8868 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:12.149129 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.149407 8868 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:12.150044 5032 raft_consensus.cc:1261] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:12.151824 8868 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:12.152189 5856 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:08:12.153367 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:12.156177 8871 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:12.159327 5036 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 04:08:12.115018 8691 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.586s user 0.005s sys 0.020s
I20260806 04:08:12.166625 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.135710 8302 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:12.177834 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:12.181972 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 58629.7us
I20260806 04:08:12.183393 4992 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.184566 5812 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.097630 8833 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:12.168359 8691 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.735s user 0.006s sys 0.020s
I20260806 04:08:12.136879 8756 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:12.180408 8302 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:12.185590 8617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.010s sys 0.030s
I20260806 04:08:12.209446 8667 consensus_queue.cc:237] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.219334 8401 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:08:12.220835 8870 log.cc:924] Time spent T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260806 04:08:12.173727 8698 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 04:08:12.224244 5037 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 04:08:12.180604 8732 consensus_queue.cc:1046] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 04:08:12.230299 8732 consensus_queue.cc:1225] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:08:12.231142 8846 consensus_queue.cc:1046] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260806 04:08:12.211580 8617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.011s sys 0.031s
I20260806 04:08:12.243026 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.188709 8664 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260806 04:08:12.243670 5032 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:12.203890 8631 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:12.257800 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.231151 8732 raft_consensus.cc:3032] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:12.400962 8726 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:12.401506 8726 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:12.397451 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 214123us
I20260806 04:08:12.422859 4992 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.425288 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:12.396548 8875 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.083s user 0.000s sys 0.002s
I20260806 04:08:12.258231 8197 raft_consensus.cc:3032] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:12.431813 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.450348 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.451200 5029 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:12.433868 4992 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.441833 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.441838 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.443953 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.452510 5035 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:12.444164 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.453059 5026 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:12.258777 8708 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:12.304484 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:12.447260 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.446882 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.456199 5032 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:12.451733 5867 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:12.399145 8732 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:12.459180 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:12.474206 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 43519.2us
I20260806 04:08:12.474689 5867 raft_consensus.cc:2379] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:12.331719 8534 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:12.476636 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.477416 8708 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:12.478327 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.443166 8197 raft_consensus.cc:729] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:12.485248 5032 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:12.486018 5035 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:12.476208 4992 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:12.442013 8871 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:12.499081 5034 raft_consensus.cc:1261] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:08:12.427160 8600 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:12.455541 8873 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:12.540410 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 64320.8us
I20260806 04:08:12.545425 4992 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.512391 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:12.563390 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 287618us
I20260806 04:08:12.516477 8631 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:12.552851 8708 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:12.585047 8520 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:12.590006 8708 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.550833 8873 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.592650 8708 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:12.594265 8873 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:12.595716 8883 log.cc:924] Time spent T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260806 04:08:12.600070 8780 raft_consensus.cc:3032] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:12.600564 8780 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:12.601529 8780 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.570775 8756 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.574493 8631 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:12.605494 8631 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:12.608893 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:45392 (request call id 448) took 1438 ms. Trace:
I20260806 04:08:12.609340 4920 rpcz_store.cc:276] 0806 04:08:11.170602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:11.171119 (+ 517us) service_pool.cc:227] Handling call
0806 04:08:12.608112 (+1436993us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54003557d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23de89fc6e 00007f23d7f96807 00007f23d7f96792 00007f23d7f966c1 00007f23d7f9663c 00007f23d7f958f2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:12.608790 (+ 678us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1610542336}
I20260806 04:08:12.610230 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.544608 8197 consensus_queue.cc:260] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.575577 8732 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.539680 8600 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:12.580099 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:08:12.619208 5863 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:12.606374 8803 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.623116 5026 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 7. Ops: []
I20260806 04:08:12.628783 8197 raft_consensus.cc:3037] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:12.628818 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.581848 8833 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.632699 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 403) took 3400 ms. Trace:
I20260806 04:08:12.633136 4927 rpcz_store.cc:276] 0806 04:08:09.231939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.272772 (+ 40833us) service_pool.cc:227] Handling call
0806 04:08:09.275136 (+ 2364us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:09.275394 (+ 258us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.275429 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:12.035238 (+2759809us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b2c0000f228. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d833de11 00007f23d834d442 00007f23d9995c11 00007f23d995a58a 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772
0806 04:08:12.035715 (+ 477us) log.cc:842] Serialized 26 byte log entry
0806 04:08:12.330029 (+294314us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:12.330084 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:12.330183 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:12.632393 (+302210us) raft_consensus.cc:1671] finished
0806 04:08:12.632413 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:12.632635 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75957504,"thread_start_us":270831,"threads_started":1,"wal-append.queue_time_us":578416}
I20260806 04:08:12.634953 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.635736 5026 raft_consensus.cc:2354] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4.
I20260806 04:08:12.636211 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.636711 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.637336 5035 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:12.637799 8780 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:12.638060 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 670) took 3551 ms. Trace:
I20260806 04:08:12.618893 8520 raft_consensus.cc:686] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:12.638756 5859 rpcz_store.cc:276] 0806 04:08:09.086101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.086226 (+ 125us) service_pool.cc:227] Handling call
0806 04:08:09.455670 (+369444us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:09.508674 (+ 53004us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:12.637838 (+3129164us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b540051ed50. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:12.637975 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":811997568}
I20260806 04:08:12.639997 8520 consensus_queue.cc:237] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:12.640916 5034 raft_consensus.cc:1261] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:08:12.618892 8779 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:12.636842 5032 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:12.644681 5029 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:12.640794 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.655140 5030 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:12.643188 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:12.664847 8391 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.665585 5860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 334) took 2808 ms. Trace:
I20260806 04:08:12.667411 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 617) took 1201 ms. Trace:
I20260806 04:08:12.667883 5862 rpcz_store.cc:276] 0806 04:08:11.465620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:11.465807 (+ 187us) service_pool.cc:227] Handling call
0806 04:08:12.667225 (+1201418us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b54001f6b54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:12.667343 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1066274560}
I20260806 04:08:12.666009 5860 rpcz_store.cc:276] 0806 04:08:09.857398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.857561 (+ 163us) service_pool.cc:227] Handling call
0806 04:08:09.921940 (+ 64379us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:10.494473 (+572533us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:12.664599 (+2170126us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54006ed054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:12.665374 (+ 775us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b54006ed050. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:12.665523 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":779985024}
I20260806 04:08:12.673918 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.676652 5026 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:12.624471 8708 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:12.684916 5032 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:12.627599 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.684755 5035 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:12.706171 5026 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:08:12.641631 8779 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.683027 8871 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:12.670706 8868 consensus_queue.cc:1046] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260806 04:08:12.722675 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.674113 8730 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260806 04:08:12.725786 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.728176 4930 raft_consensus.cc:1261] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260806 04:08:12.728667 4930 pending_rounds.cc:76] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Aborting all ops after (but not including) 7
I20260806 04:08:12.728912 4930 pending_rounds.cc:97] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260806 04:08:12.729192 4930 raft_consensus.cc:2866] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:08:12.732113 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 268676us
I20260806 04:08:12.720254 8302 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:12.704092 8708 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:12.736405 5812 leader_election.cc:334] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.553s (SENT)
I20260806 04:08:12.741250 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:12.742769 8812 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 04:08:12.740072 8862 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.024s sys 0.023s
W20260806 04:08:12.743587 8862 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s user 0.024s sys 0.023s
I20260806 04:08:12.744166 8708 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:12.746220 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.747851 8779 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:12.711145 8726 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:12.752815 8887 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260806 04:08:12.754400 8726 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.756454 5026 raft_consensus.cc:1226] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:12.759671 5035 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [5.6-6.7]
I20260806 04:08:12.759788 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.762669 5032 raft_consensus.cc:1226] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:12.764810 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.765799 5032 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.771725 5038 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
W20260806 04:08:12.774041 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.69456e+06us
I20260806 04:08:12.774330 8814 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:12.785346 8814 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.774585 8631 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:12.788030 8814 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:12.788604 8631 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.789692 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.789877 5030 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:12.790151 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.790326 5029 raft_consensus.cc:2362] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 2. Re-sending same reply.
I20260806 04:08:12.791051 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.792023 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.791169 5032 raft_consensus.cc:2354] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4.
I20260806 04:08:12.802879 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.804281 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.804628 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.805471 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.806229 5034 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.769621 8197 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260806 04:08:12.805497 5037 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:12.807008 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.807783 5032 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.807586 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.808238 5029 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:12.808554 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.809047 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.809643 5030 raft_consensus.cc:1226] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:12.809690 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.809198 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.811537 5032 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:12.810621 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.810520 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.813053 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.813592 5030 raft_consensus.cc:3037] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:12.820820 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:12.821489 4920 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:12.823799 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:12.824741 4813 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
W20260806 04:08:12.828962 4669 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56998 (request call id 143) took 18127 ms (client timeout 15000 ms). Trace:
I20260806 04:08:12.629006 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:08:12.832276 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:12.833154 4923 raft_consensus.cc:2379] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:12.842918 5035 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:12.851028 5862 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:08:12.852415 4816 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:08:12.852723 5859 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:08:12.859031 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.868119 5034 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.871316 5037 raft_consensus.cc:1226] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:12.876605 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:12.877905 4923 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
W20260806 04:08:12.882417 4768 leader_election.cc:334] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.512s (SENT)
I20260806 04:08:12.883255 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:12.889377 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:12.890249 4926 raft_consensus.cc:1226] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:12.888490 5032 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:12.900000 4923 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 04:08:12.901290 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.903182 5029 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:08:12.907688 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.911553 4920 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:12.913566 5034 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 7: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.893795 5030 raft_consensus.cc:2454] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
W20260806 04:08:12.916401 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 234564us
I20260806 04:08:12.918453 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:12.920027 4768 leader_election.cc:334] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.526s (SENT)
I20260806 04:08:12.870508 8600 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:12.920897 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:12.869668 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.925258 5032 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.927961 4926 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 04:08:12.929033 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:12.932314 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.937364 4876 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:12.932403 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:12.938879 5812 leader_election.cc:334] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.974s (SENT)
I20260806 04:08:12.939064 4813 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:12.931437 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.941970 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:12.946388 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:12.894937 8871 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:12.951771 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.955483 5857 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:12.958772 8399 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:12.962939 8302 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:12.963265 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.964860 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.965554 5037 raft_consensus.cc:2362] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8. Re-sending same reply.
I20260806 04:08:12.883597 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:12.970973 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:12.979848 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 59871.9us
I20260806 04:08:12.967334 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.966616 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:12.921653 8779 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:12.982122 5037 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:12.968072 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:12.982930 4816 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:12.978448 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:12.970707 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:12.985278 5030 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:12.985666 4768 connection.cc:232] RPC call timeout handler was delayed by -1.6657364659477025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
I20260806 04:08:12.986244 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:12.986919 4768 leader_election.cc:334] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.580s (SENT)
I20260806 04:08:12.987870 5034 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:12.987699 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:12.987068 5030 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:13.013985 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.019532 4925 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:12.962925 8391 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:13.022284 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.031637 4817 raft_consensus.cc:2379] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 10: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:13.029525 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:13.033278 5036 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:13.033383 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.034435 4816 raft_consensus.cc:2396] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 04:08:13.036005 5034 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:12.966828 8600 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:13.017091 8164 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:13.038973 5037 raft_consensus.cc:1261] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:08:13.014865 8814 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:13.099505 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.100603 4818 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:12.968501 8833 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
I20260806 04:08:13.094123 5032 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 5. Ops: []
I20260806 04:08:12.989326 8281 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:08:13.029165 8399 consensus_queue.cc:1046] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
I20260806 04:08:13.029970 8302 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:13.107065 8903 log.cc:924] Time spent T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 04:08:12.979339 8871 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.105180 4813 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:13.112144 4996 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:13.113489 5037 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:13.121681 5036 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 9.
I20260806 04:08:13.123821 5028 raft_consensus.cc:1226] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 1. Current term is 2. Ops: []
I20260806 04:08:13.130470 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.130738 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:13.130625 5036 raft_consensus.cc:1226] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
W20260806 04:08:13.131011 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 997e4d9ff3ca465199a888e8af23ed7b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:13.114434 8846 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:13.127277 8399 consensus_queue.cc:1225] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer 99690415a7b3441fbdb5e943667556a4 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:08:13.129521 8281 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:13.144495 8846 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:13.124552 8814 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:13.128456 8302 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:12.829540 4669 rpcz_store.cc:269] 0806 04:07:54.701474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:54.701739 (+ 265us) service_pool.cc:227] Handling call
0806 04:08:12.828670 (+18126931us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:07:59.662472 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:07:59.879667 (+217195us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:07:59.892865 (+ 13198us) write_op.cc:435] Acquired schema lock
0806 04:07:59.903985 (+ 11120us) tablet.cc:660] Decoding operations
0806 04:08:00.547091 (+643106us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:08:00.573115 (+ 26024us) write_op.cc:641] Partition lock acquired for write op
0806 04:08:00.576348 (+ 3233us) tablet.cc:683] Acquiring locks for 9 operations
0806 04:08:00.615552 (+ 39204us) tablet.cc:699] Row locks acquired
0806 04:08:00.615578 (+ 26us) write_op.cc:260] PREPARE: finished
0806 04:08:00.615734 (+ 156us) write_op.cc:270] Start()
0806 04:08:00.615875 (+ 141us) write_op.cc:276] Timestamp: P: 1785989280615681 usec, L: 0
0806 04:08:00.615907 (+ 32us) op_driver.cc:331] REPLICATION: starting
0806 04:08:00.641999 (+ 26092us) log.cc:842] Serialized 3570 byte log entry
0806 04:08:03.206010 (+2564011us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d6a145fe 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb
0806 04:08:06.567238 (+3361228us) op_driver.cc:444] REPLICATION: finished
0806 04:08:06.752384 (+185146us) write_op.cc:301] APPLY: starting
0806 04:08:06.759778 (+ 7394us) tablet.cc:1365] starting BulkCheckPresence
0806 04:08:06.837861 (+ 78083us) tablet.cc:1368] finished BulkCheckPresence
0806 04:08:06.837884 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:08:06.839518 (+ 1634us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:08:06.839923 (+ 405us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0806 04:08:06.839968 (+ 45us) write_op.cc:312] APPLY: finished
0806 04:08:06.840402 (+ 434us) log.cc:842] Serialized 71 byte log entry
0806 04:08:06.841343 (+ 941us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:08:06.841595 (+ 252us) write_op.cc:454] Released schema lock
0806 04:08:06.841694 (+ 99us) write_op.cc:341] FINISH: Updating metrics
0806 04:08:06.842157 (+ 463us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:08:06.842200 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":123505,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":170235,"apply.run_cpu_time_us":4742,"apply.run_wall_time_us":90322,"num_ops":9,"prepare.queue_time_us":72761,"prepare.run_cpu_time_us":49799,"prepare.run_wall_time_us":5873966,"raft.queue_time_us":1047893,"raft.run_cpu_time_us":20034,"raft.run_wall_time_us":284268,"replication_time_us":5851674,"spinlock_wait_cycles":11027840,"thread_start_us":5973477,"threads_started":4,"wal-append.queue_time_us":4748306,"wal-append.run_cpu_time_us":30714,"wal-append.run_wall_time_us":2450798}]]}
I20260806 04:08:13.215773 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:13.173780 8917 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:13.179523 8730 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260806 04:08:13.210937 8890 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 04:08:13.181083 8846 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:13.228549 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 398601us
W20260806 04:08:13.227165 8904 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20260806 04:08:13.222361 8917 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:13.187839 8906 consensus_queue.cc:1046] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
W20260806 04:08:13.253329 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 234281us
I20260806 04:08:13.261392 4876 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:13.265168 4876 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:13.267575 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 280664us
W20260806 04:08:13.268530 4768 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.581s (SENT)
W20260806 04:08:13.254633 8905 log.cc:924] Time spent T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20260806 04:08:13.269150 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:13.244951 8391 raft_consensus.cc:686] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:13.277019 5032 raft_consensus.cc:1226] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:13.250196 8917 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:13.277770 4876 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:13.283088 8302 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:13.283527 8242 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:13.284687 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:13.284117 8242 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:08:13.295351 8780 consensus_meta.cc:287] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.622s user 0.008s sys 0.005s
I20260806 04:08:13.272302 8814 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:13.266839 8619 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.038s
I20260806 04:08:13.271581 8722 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260806 04:08:13.303504 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:13.479022 5032 raft_consensus.cc:2354] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:13.303879 5026 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:13.502784 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 433) took 1071 ms. Trace:
I20260806 04:08:13.503525 4919 rpcz_store.cc:276] 0806 04:08:12.431517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.432096 (+ 579us) service_pool.cc:227] Handling call
0806 04:08:12.449230 (+ 17134us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:12.463801 (+ 14571us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:12.463852 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:12.476636 (+ 12784us) log.cc:842] Serialized 52 byte log entry
0806 04:08:12.711360 (+234724us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:12.711413 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:12.711509 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:13.368782 (+657273us) raft_consensus.cc:1671] finished
0806 04:08:13.368809 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:13.502683 (+133874us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":503,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":28107,"thread_start_us":234219,"threads_started":1,"wal-append.queue_time_us":456805}
W20260806 04:08:13.352730 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 222073us
W20260806 04:08:13.423416 8923 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260806 04:08:13.504314 8302 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:13.428974 8943 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20260806 04:08:13.338815 8875 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:08:13.270758 8833 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:13.420331 8935 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:13.627857 8833 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:13.326534 8281 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.339s
I20260806 04:08:13.285746 8872 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:13.667109 8913 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:13.307185 8399 raft_consensus.cc:2781] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:13.667701 8913 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.672999 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:13.674008 5035 raft_consensus.cc:2354] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 9. Current term is 10.
I20260806 04:08:13.634760 8302 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:13.678231 8938 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
I20260806 04:08:13.609979 8391 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:13.340832 8871 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:13.735724 8871 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:13.736547 8871 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.758912 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:13.760910 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:13.761569 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 468143us
W20260806 04:08:13.754513 4879 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.009s sys 0.020s
W20260806 04:08:13.766170 4879 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.010s sys 0.020s
W20260806 04:08:13.766072 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 497544us
W20260806 04:08:13.766688 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 796030us
W20260806 04:08:13.766978 4768 connection.cc:232] RPC call timeout handler was delayed by -1.6072364659048617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260806 04:08:13.773181 4768 connection.cc:232] RPC call timeout handler was delayed by -1.606336466036737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
W20260806 04:08:13.775018 4768 leader_election.cc:334] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.614s (SENT)
I20260806 04:08:13.775700 4768 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:13.773002 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.777933 5036 raft_consensus.cc:2379] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:13.782068 5036 raft_consensus.cc:1226] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:13.764878 8868 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:13.724836 8833 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:13.784915 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.002s sys 0.014s
I20260806 04:08:13.785030 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:13.785635 4768 connection.cc:232] RPC call timeout handler was delayed by -1.5748364659957588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
W20260806 04:08:13.789271 4768 connection.cc:232] RPC call timeout handler was delayed by -1.5073364658746868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18830000000000002s for a response
W20260806 04:08:13.789955 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4686364659573883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20260806 04:08:13.790504 4768 connection.cc:232] RPC call timeout handler was delayed by -1.4596364658791572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260806 04:08:13.770900 8942 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:13.798616 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:13.790877 4768 connection.cc:232] RPC call timeout handler was delayed by -1.3993364661000669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20030000000000003s for a response
W20260806 04:08:13.800642 4768 connection.cc:232] RPC call timeout handler was delayed by -1.2814364659134299s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260806 04:08:13.801321 4768 connection.cc:232] RPC call timeout handler was delayed by -1.064936030190438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
W20260806 04:08:13.792557 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.002s sys 0.014s
W20260806 04:08:13.802136 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 776680us
I20260806 04:08:13.778436 8302 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:13.777460 8399 raft_consensus.cc:686] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:13.805145 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.814164 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.809384 4768 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:13.809494 8724 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 4.297s user 0.009s sys 0.013s
W20260806 04:08:13.820258 8724 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 4.312s user 0.010s sys 0.014s
W20260806 04:08:13.820325 8164 consensus_meta.cc:287] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.706s user 0.012s sys 0.008s
I20260806 04:08:13.820843 8164 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:13.808369 8619 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:13.823503 8619 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:13.810684 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.811709 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.817713 4805 raft_consensus.cc:1226] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:13.826007 8619 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:13.827548 8399 consensus_queue.cc:237] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.828598 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:13.829402 5032 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:13.832227 8456 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:13.832891 8456 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:13.834965 8815 raft_consensus.cc:3032] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:13.835457 8815 raft_consensus.cc:729] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:13.836462 8815 consensus_queue.cc:260] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.819011 8928 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:13.791529 8730 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:13.850802 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.869632 5032 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:13.776363 8936 consensus_queue.cc:1035] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.106s
I20260806 04:08:13.876629 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.798023 8758 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:13.822427 8589 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:13.885774 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.885309 8928 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:13.901093 8928 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:13.902891 8242 consensus_meta.cc:287] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.601s user 0.006s sys 0.015s
I20260806 04:08:13.903033 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.903928 5036 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:13.909008 8708 raft_consensus.cc:3032] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:13.909520 8708 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:13.886397 4804 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:13.912904 4804 raft_consensus.cc:1226] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:13.887063 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.889209 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.789803 8944 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:13.876804 4818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:08:13.915727 4818 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:08:13.916157 4818 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:08:13.892278 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:13.888458 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 79177us
W20260806 04:08:13.891053 8899 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.007s sys 0.028s
I20260806 04:08:13.897178 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.898006 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.925625 4809 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:13.903455 8242 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.918341 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:13.926498 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.013s sys 0.032s
W20260806 04:08:13.927268 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.013s sys 0.032s
I20260806 04:08:13.893520 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.927750 4805 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
W20260806 04:08:13.927527 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 674574us
I20260806 04:08:13.816339 8942 raft_consensus.cc:686] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:13.892599 8730 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:13.938830 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:13.939735 5856 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:13.940666 8242 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:13.941479 8730 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:13.941419 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:13.932461 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:13.924520 8899 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.007s sys 0.029s
W20260806 04:08:13.945209 8781 log.cc:924] Time spent T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 3.432s user 0.000s sys 0.013s
I20260806 04:08:13.910527 8708 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.934278 8935 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:13.800266 8872 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.129s
I20260806 04:08:13.947196 4995 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:13.946255 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:13.947536 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 431) took 1520 ms. Trace:
I20260806 04:08:13.948706 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:13.911342 8281 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:13.929226 8871 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:13.959679 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 191093us
I20260806 04:08:13.961202 4930 rpcz_store.cc:276] 0806 04:08:12.427133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.427847 (+ 714us) service_pool.cc:227] Handling call
0806 04:08:12.428167 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:12.729637 (+301470us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:13.947285 (+1217648us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54003543d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:13.947476 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688831360}
I20260806 04:08:13.931981 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:13.980852 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 49505.2us
I20260806 04:08:13.983021 4882 leader_election.cc:398] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7.
I20260806 04:08:13.983608 4882 leader_election.cc:401] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:13.913693 8758 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:13.979796 5036 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:13.987147 5032 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:13.988902 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:13.959583 8906 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.004s sys 0.047s
I20260806 04:08:13.993110 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:14.005425 4882 connection.cc:232] RPC call timeout handler was delayed by -1.4197928390931338s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20260806 04:08:14.006028 4882 connection.cc:232] RPC call timeout handler was delayed by -1.4080928391776979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260806 04:08:14.006456 4882 connection.cc:232] RPC call timeout handler was delayed by -1.3864928390830755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20260806 04:08:14.006975 4882 connection.cc:232] RPC call timeout handler was delayed by -1.3261928390711546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1615s for a response
W20260806 04:08:14.007388 4882 connection.cc:232] RPC call timeout handler was delayed by -1.2811928391456604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20260806 04:08:14.007239 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:13.979130 8730 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:13.967621 8899 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:13.985028 8708 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:14.014086 4808 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
W20260806 04:08:13.995565 8906 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.004s sys 0.047s
I20260806 04:08:14.016474 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2240) took 4306 ms. Trace:
I20260806 04:08:14.017482 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:14.010481 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 233213us
I20260806 04:08:13.992295 8709 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:13.996074 8705 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:14.019230 4876 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:14.020890 5036 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:14.019927 4882 connection.cc:232] RPC call timeout handler was delayed by -1.2235928392037749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260806 04:08:14.028721 4882 connection.cc:232] RPC call timeout handler was delayed by -1.2082928391173482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
W20260806 04:08:14.029150 4882 connection.cc:232] RPC call timeout handler was delayed by -1.0984928391408175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
W20260806 04:08:14.029603 4882 connection.cc:232] RPC call timeout handler was delayed by -1.0633928391616791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260806 04:08:14.030351 4882 connection.cc:232] RPC call timeout handler was delayed by -1.0264928392134607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1948s for a response
I20260806 04:08:14.016966 4810 rpcz_store.cc:276] 0806 04:08:09.706258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.710126 (+ 3868us) service_pool.cc:227] Handling call
0806 04:08:09.710565 (+ 439us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:09.721697 (+ 11132us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.721748 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:09.994621 (+272873us) log.cc:842] Serialized 26 byte log entry
0806 04:08:10.057725 (+ 63104us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:10.057802 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:10.057951 (+ 149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:13.944457 (+3886506us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:13.994472 (+ 50015us) raft_consensus.cc:1671] finished
0806 04:08:13.994501 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:14.012213 (+ 17712us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10592,"raft.queue_time_us":56508,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8396,"spinlock_wait_cycles":9309056,"thread_start_us":28056,"threads_started":1,"wal-append.queue_time_us":483548}
I20260806 04:08:14.005704 8958 raft_consensus.cc:3037] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:08:13.999533 8913 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:14.014716 8730 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:14.088949 4876 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:14.085289 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:14.082875 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 124419us
I20260806 04:08:13.960857 8944 consensus_queue.cc:1046] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260806 04:08:14.094571 4876 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:14.097385 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:14.099876 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:14.106025 5036 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:14.028030 8705 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:14.114796 4876 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:13.997907 8940 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.098s
I20260806 04:08:14.113710 4818 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.125028 4818 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:14.120731 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.128000 8619 consensus_queue.cc:1046] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:14.086550 8456 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:14.085918 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:14.094226 8730 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:14.139115 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.104463 8399 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:14.136276 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:14.136482 4876 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:14.139971 5036 raft_consensus.cc:2354] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:08:14.088496 8200 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:14.133026 8391 consensus_queue.cc:1035] T faa361eb5ab5463895c3f7af5e87237c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260806 04:08:14.057731 8709 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:14.120204 8705 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:14.116863 8936 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:14.150179 8281 raft_consensus.cc:686] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:14.149335 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.148928 4876 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:14.122145 8959 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:14.027282 8899 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:14.135663 8935 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:14.135982 8814 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:14.187408 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.141714 8399 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:14.195449 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:14.160415 8930 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:08:14.198413 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 74921.3us
I20260806 04:08:14.186113 8748 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:14.027700 8942 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:14.184151 8200 raft_consensus.cc:686] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:14.187731 8936 raft_consensus.cc:686] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:14.212627 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 63786us
W20260806 04:08:14.203022 8917 consensus_meta.cc:287] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.877s user 0.018s sys 0.008s
I20260806 04:08:14.215184 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:14.221129 8758 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:14.221271 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.222108 5035 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:08:14.224612 8852 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 2.608s user 0.000s sys 0.010s
I20260806 04:08:14.224918 8709 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:14.217944 4818 raft_consensus.cc:1226] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
W20260806 04:08:14.202118 8846 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.973s user 0.002s sys 0.020s
I20260806 04:08:14.197588 8399 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:14.203137 8930 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:14.228613 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 283) took 1470 ms. Trace:
I20260806 04:08:14.239493 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.239660 5026 rpcz_store.cc:276] 0806 04:08:12.758532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.759451 (+ 919us) service_pool.cc:227] Handling call
0806 04:08:14.228383 (+1468932us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b54000cec54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:14.228535 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1249035136}
I20260806 04:08:14.227356 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.271847 5028 raft_consensus.cc:2379] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:14.272367 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:14.241331 8846 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:14.274258 8971 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260806 04:08:14.208961 8814 raft_consensus.cc:686] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:14.279968 8758 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:14.213167 5856 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:14.286823 8930 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:14.231091 8917 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.288727 8281 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:14.290761 5038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2068) took 1889 ms (client timeout 1665 ms). Trace:
W20260806 04:08:14.304893 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 64654.2us
W20260806 04:08:14.394851 5038 rpcz_store.cc:269] 0806 04:08:12.401466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.424396 (+ 22930us) service_pool.cc:227] Handling call
0806 04:08:14.290473 (+1866077us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54003fc5d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:14.290677 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":733837952}
I20260806 04:08:14.290977 8935 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.225737 8589 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:14.372568 8899 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:14.338133 4923 raft_consensus.cc:3032] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:14.423185 4923 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:14.341645 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 249035us
I20260806 04:08:14.320791 8463 raft_consensus.cc:3032] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:14.322309 8868 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:14.333447 8981 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:14.351095 8962 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:14.358666 8200 consensus_queue.cc:237] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.338776 8959 raft_consensus.cc:686] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:14.242619 5035 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 1.
I20260806 04:08:14.383996 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.362953 8975 raft_consensus.cc:3032] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:14.366339 8748 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:14.396152 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.403406 4882 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:14.410576 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.337085 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.417393 8814 consensus_queue.cc:237] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.391928 8758 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.421478 8917 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:14.366150 8944 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:14.453147 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.423981 4923 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:14.458505 4768 connection.cc:232] RPC call timeout handler was delayed by -1.6339704007841647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
W20260806 04:08:14.468607 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 289113us
I20260806 04:08:14.467247 5038 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
W20260806 04:08:14.474509 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 71257.5us
I20260806 04:08:14.372074 8730 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:14.505895 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 291669us
I20260806 04:08:14.503567 5035 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:14.519011 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.509204 4923 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:14.354534 8936 consensus_queue.cc:237] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.511173 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.512084 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:14.428527 8463 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:14.515515 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:14.495638 4768 connection.cc:232] RPC call timeout handler was delayed by -1.5492594379466027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.205s for a response
I20260806 04:08:14.519169 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:14.527534 5812 leader_election.cc:334] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.740s (SENT)
I20260806 04:08:14.454468 8975 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:14.549521 5035 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:14.551203 5812 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:14.555859 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.556851 4808 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260806 04:08:14.557017 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.551891 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.557888 4814 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:14.559768 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.563163 5026 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:14.564495 5037 raft_consensus.cc:2354] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 9. Current term is 10.
I20260806 04:08:14.567549 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:14.568986 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:14.577811 4768 leader_election.cc:334] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.506s (SENT)
I20260806 04:08:14.578524 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:14.583935 4810 raft_consensus.cc:2379] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 2: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:14.583760 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:14.569756 5035 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:14.429208 8589 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:14.465611 8868 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.468240 8977 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.134s
I20260806 04:08:14.494875 8997 consensus_queue.cc:1046] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.167s
I20260806 04:08:14.507382 8944 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:14.473500 8962 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:14.473752 8981 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:15.018345 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.007s sys 0.031s
W20260806 04:08:15.119743 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.007s sys 0.036s
W20260806 04:08:15.044193 5036 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.021s sys 0.042s
W20260806 04:08:15.123991 5036 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.676s user 0.022s sys 0.042s
I20260806 04:08:14.697228 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2082) took 1679 ms. Trace:
I20260806 04:08:14.629628 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.585551 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:14.589805 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:15.128322 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:15.061301 4879 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.009s sys 0.007s
W20260806 04:08:15.129019 4879 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.703s user 0.010s sys 0.008s
I20260806 04:08:14.590708 5038 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:14.919028 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:14.586846 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:15.120119 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 542313us
W20260806 04:08:15.133450 4768 leader_election.cc:334] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.536s (SENT)
W20260806 04:08:15.137745 4768 leader_election.cc:334] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.536s (SENT)
I20260806 04:08:15.134312 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:15.128646 5035 raft_consensus.cc:2396] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 04:08:15.142681 5026 raft_consensus.cc:2379] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:15.143285 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:15.144419 4808 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:15.127094 5034 rpcz_store.cc:276] 0806 04:08:13.017959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:13.036996 (+ 19037us) service_pool.cc:227] Handling call
0806 04:08:13.096110 (+ 59114us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:13.115006 (+ 18896us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:13.124399 (+ 9393us) log.cc:842] Serialized 14 byte log entry
0806 04:08:13.912814 (+788415us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b18001f34e0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a14345 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e157b 00007f23d996488a
0806 04:08:14.078165 (+165351us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:14.091191 (+ 13026us) log.cc:842] Serialized 26 byte log entry
0806 04:08:14.091384 (+ 193us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:14.342342 (+250958us) log.cc:842] Serialized 14 byte log entry
0806 04:08:14.342614 (+ 272us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:14.342790 (+ 176us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:14.696837 (+354047us) raft_consensus.cc:1671] finished
0806 04:08:14.696873 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:14.697104 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18384,"prepare.queue_time_us":441849,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":83,"raft.queue_time_us":70325,"raft.run_cpu_time_us":595,"raft.run_wall_time_us":47725,"spinlock_wait_cycles":6569856,"thread_start_us":400643,"threads_started":2,"wal-append.queue_time_us":1005054}
I20260806 04:08:14.548795 8709 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:15.128680 4809 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
W20260806 04:08:15.120311 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 620256us
W20260806 04:08:15.150640 4876 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.932s user 0.010s sys 0.004s
W20260806 04:08:15.151075 4876 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.932s user 0.010s sys 0.004s
W20260806 04:08:14.717257 8833 consensus_meta.cc:287] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.917s user 0.008s sys 0.007s
I20260806 04:08:15.149537 4813 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:15.157562 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:15.158483 4816 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:14.554548 8917 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:15.155949 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:14.566821 8629 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:15.166791 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:15.167765 5038 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
W20260806 04:08:14.901366 8996 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.408s user 0.000s sys 0.001s
I20260806 04:08:15.150213 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:14.755684 8958 consensus_meta.cc:287] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.649s user 0.009s sys 0.003s
W20260806 04:08:15.125586 8793 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.001s sys 0.010s
I20260806 04:08:14.584131 8942 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:15.162982 4882 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:15.044903 8952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.025s sys 0.042s
I20260806 04:08:14.583503 8731 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 04:08:14.784230 8971 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 04:08:14.610450 8391 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:15.153565 8708 consensus_meta.cc:287] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.132s user 0.010s sys 0.006s
I20260806 04:08:15.098174 8758 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:15.081799 8730 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:15.168907 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:15.045253 8954 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
W20260806 04:08:15.169409 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.003s sys 0.011s
I20260806 04:08:15.189996 9015 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:15.304935 8958 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7.
W20260806 04:08:15.176930 8399 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.889s user 0.010s sys 0.003s
I20260806 04:08:14.673404 8748 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.178126 8534 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:15.179605 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1699) took 4319 ms. Trace:
I20260806 04:08:15.018699 8846 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:15.157081 8756 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:15.158763 9007 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:15.191519 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:15.191789 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:15.553421 5026 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:15.278637 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:15.554275 5035 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:15.308802 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 145912us
I20260806 04:08:14.699743 8991 consensus_queue.cc:1046] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.609s
W20260806 04:08:15.282685 8793 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.001s sys 0.011s
I20260806 04:08:15.152454 8709 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:14.744831 8975 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.358067 5028 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:15.359817 5038 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:15.310719 8952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s user 0.027s sys 0.043s
W20260806 04:08:15.364974 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.004s sys 0.011s
I20260806 04:08:15.165359 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:15.168816 8833 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.171144 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:15.177846 8112 consensus_queue.cc:1046] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:14.660327 8959 consensus_queue.cc:237] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.310364 8640 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 04:08:15.442142 4995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.066s sys 0.168s
W20260806 04:08:15.575781 4995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.811s user 0.067s sys 0.173s
W20260806 04:08:15.576507 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 411234us
I20260806 04:08:15.577630 4876 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:15.582998 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:15.440663 4928 rpcz_store.cc:276] 0806 04:08:10.860410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:10.877757 (+ 17347us) service_pool.cc:227] Handling call
0806 04:08:10.903767 (+ 26010us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:11.605915 (+702148us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:15.179257 (+3573342us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54003efb54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:15.179494 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1613213568}
I20260806 04:08:15.560775 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:15.586336 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:15.587340 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1739) took 1165 ms. Trace:
I20260806 04:08:15.587746 5035 rpcz_store.cc:276] 0806 04:08:14.422211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.586006 (+1163795us) service_pool.cc:227] Handling call
0806 04:08:15.587276 (+ 1270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:15.590109 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:15.595418 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:15.171545 8917 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:15.567673 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1728) took 1287 ms. Trace:
I20260806 04:08:15.625017 5038 rpcz_store.cc:276] 0806 04:08:14.279923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.168589 (+888666us) service_pool.cc:227] Handling call
0806 04:08:15.567588 (+398999us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:15.570116 5037 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:08:15.626505 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1735) took 1238 ms. Trace:
I20260806 04:08:15.627014 5037 rpcz_store.cc:276] 0806 04:08:14.388431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.151552 (+763121us) service_pool.cc:227] Handling call
0806 04:08:15.626426 (+474874us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:15.629978 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:15.631086 4920 raft_consensus.cc:2379] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:15.566767 4805 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.056s sys 0.093s
I20260806 04:08:15.624153 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:15.632897 4805 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.049s user 0.056s sys 0.093s
I20260806 04:08:15.648759 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:15.649794 4920 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:15.315027 8942 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.559234 8708 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:15.699843 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:15.357913 8391 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:15.699771 4928 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:15.698153 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:15.634620 8705 consensus_meta.cc:287] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.450s user 0.014s sys 0.000s
I20260806 04:08:15.751400 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:15.768929 5038 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:08:15.770010 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 192062us
I20260806 04:08:15.548588 8534 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:15.425379 8954 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:15.752554 4770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.68:38873 timed out after 1.777s (SENT) [suppressed 37 similar messages]
I20260806 04:08:15.566449 9000 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:15.439073 8399 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.767128 4920 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:15.770936 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:15.796751 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:15.802500 4771 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.032s user 0.114s sys 0.185s
I20260806 04:08:15.754508 4925 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
W20260806 04:08:15.753690 9012 log.cc:924] Time spent T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.330s user 0.001s sys 0.001s
I20260806 04:08:15.556280 8756 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:15.934072 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.020s sys 0.016s
I20260806 04:08:15.930492 5037 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:15.931025 8995 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:15.931016 5035 raft_consensus.cc:2354] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5.
W20260806 04:08:15.937989 4818 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.000s sys 0.057s
I20260806 04:08:15.698167 8977 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:15.751307 8731 raft_consensus.cc:2741] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:08:15.937412 8730 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:16.027350 8730 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:16.028057 8833 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:16.028918 8399 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:15.948686 8589 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:15.867275 8705 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:16.029927 8917 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:16.030000 8730 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:15.849713 9024 consensus_queue.cc:1046] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.274s
I20260806 04:08:15.776818 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:15.994293 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.829s user 0.020s sys 0.016s
I20260806 04:08:15.999205 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1729) took 1718 ms. Trace:
W20260806 04:08:16.013520 9019 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.589s user 0.000s sys 0.000s
W20260806 04:08:16.015229 9011 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.449s user 0.000s sys 0.000s
W20260806 04:08:16.016964 9022 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.327s user 0.000s sys 0.000s
W20260806 04:08:16.017584 8997 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.000s sys 0.000s
W20260806 04:08:16.024024 9016 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.704s user 0.000s sys 0.001s
I20260806 04:08:16.044418 5034 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:15.998605 9017 raft_consensus.cc:3032] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:08:16.048838 5035 rpcz_store.cc:276] 0806 04:08:14.280369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.767891 (+1487522us) service_pool.cc:227] Handling call
0806 04:08:15.999106 (+231215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.267613 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:16.268416 5035 raft_consensus.cc:2362] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7. Re-sending same reply.
I20260806 04:08:16.269003 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1745) took 1410 ms. Trace:
I20260806 04:08:16.269357 5035 rpcz_store.cc:276] 0806 04:08:14.858334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.267242 (+1408908us) service_pool.cc:227] Handling call
0806 04:08:16.268938 (+ 1696us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:16.048815 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 902003us
I20260806 04:08:16.031811 8705 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:16.054370 8983 log.cc:924] Time spent T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 1.775s user 0.004s sys 0.013s
I20260806 04:08:16.058465 5863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 343) took 3596 ms. Trace:
I20260806 04:08:16.062917 8692 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:16.063303 8981 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:16.063623 8709 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:15.771783 8708 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:15.996527 8944 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:16.008147 8868 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:15.564529 9007 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:16.168315 9006 log.cc:924] Time spent T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.323s user 0.000s sys 0.000s
I20260806 04:08:16.251294 8962 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:16.270396 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:16.294760 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.687s user 0.005s sys 0.044s
I20260806 04:08:16.068011 5854 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:16.298497 5863 rpcz_store.cc:276] 0806 04:08:12.461951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.462093 (+ 142us) service_pool.cc:227] Handling call
0806 04:08:12.615459 (+153366us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:12.621681 (+ 6222us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:16.058181 (+3436500us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400004e50. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:16.058383 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":351642368}
W20260806 04:08:16.229899 8997 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.788s user 0.000s sys 0.000s
W20260806 04:08:16.363351 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.756s user 0.006s sys 0.045s
I20260806 04:08:16.266642 9017 raft_consensus.cc:729] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:16.383587 5035 raft_consensus.cc:2379] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:16.392890 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1747) took 1253 ms. Trace:
I20260806 04:08:16.393515 5035 rpcz_store.cc:276] 0806 04:08:15.139650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.270098 (+1130448us) service_pool.cc:227] Handling call
0806 04:08:16.392763 (+122665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.395072 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:16.396301 5035 raft_consensus.cc:2354] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5.
I20260806 04:08:16.059847 8977 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:16.447008 5034 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:16.448700 8708 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:16.449791 9007 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:16.450417 8391 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:16.452165 8942 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:16.551121 9029 log.cc:924] Time spent T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.447s user 0.000s sys 0.001s
W20260806 04:08:16.552138 8998 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.445s user 0.000s sys 0.002s
W20260806 04:08:16.404531 9022 log.cc:924] Time spent T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260806 04:08:16.393366 9012 log.cc:924] Time spent T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20260806 04:08:16.487795 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.006s sys 0.012s
W20260806 04:08:16.519773 8814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.004s sys 0.014s
W20260806 04:08:16.556085 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.006s sys 0.012s
W20260806 04:08:16.558840 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 626045us
W20260806 04:08:16.558377 8814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.936s user 0.005s sys 0.015s
W20260806 04:08:16.617951 9066 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20260806 04:08:16.590770 8954 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:16.340445 8692 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:16.621047 8930 consensus_meta.cc:287] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 2.196s user 0.013s sys 0.004s
I20260806 04:08:16.649837 8692 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:16.653241 8930 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:16.552001 9017 consensus_queue.cc:260] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:16.655277 9017 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:16.652058 8977 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:16.658546 8930 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:16.628041 8936 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.029s user 0.014s sys 0.049s
W20260806 04:08:16.663074 8936 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.084s user 0.015s sys 0.049s
W20260806 04:08:16.627545 8952 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.000s sys 0.009s
W20260806 04:08:16.664592 8952 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.665s user 0.000s sys 0.010s
W20260806 04:08:15.926952 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.000s sys 0.036s
W20260806 04:08:16.723433 4769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.014s sys 0.022s
W20260806 04:08:16.724153 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.579s user 0.000s sys 0.036s
W20260806 04:08:16.724335 4769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.560s user 0.015s sys 0.022s
W20260806 04:08:16.726634 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.58877e+06us
W20260806 04:08:16.729655 4768 leader_election.cc:334] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.707s (SENT)
I20260806 04:08:16.730618 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:16.734189 4768 leader_election.cc:334] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.714s (SENT)
I20260806 04:08:16.735149 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:16.737869 4768 leader_election.cc:334] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.772s (SENT)
W20260806 04:08:15.931203 4771 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.161s user 0.115s sys 0.185s
W20260806 04:08:16.738996 4770 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.959s user 0.007s sys 0.032s
W20260806 04:08:16.744611 4770 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.965s user 0.008s sys 0.032s
W20260806 04:08:16.744951 4768 leader_election.cc:334] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.773s (SENT)
W20260806 04:08:16.745291 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 992708us
W20260806 04:08:16.748680 4768 leader_election.cc:334] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.808s (SENT)
I20260806 04:08:16.749413 4768 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:16.739285 4805 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.002s sys 0.016s
W20260806 04:08:15.999408 4818 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.729s user 0.001s sys 0.057s
W20260806 04:08:16.751418 4878 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.241s user 0.023s sys 0.024s
I20260806 04:08:16.753063 4816 raft_consensus.cc:1261] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260806 04:08:16.756785 4919 raft_consensus.cc:2379] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:15.992278 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:16.830921 8708 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:16.832635 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1565) took 1079 ms. Trace:
I20260806 04:08:16.832433 8708 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:16.833217 4924 rpcz_store.cc:276] 0806 04:08:15.753088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.770751 (+ 17663us) service_pool.cc:227] Handling call
0806 04:08:16.832317 (+1061566us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:16.833925 8534 consensus_meta.cc:287] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.894s user 0.006s sys 0.009s
I20260806 04:08:16.040302 4927 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:16.019806 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:16.758082 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:16.758599 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 441) took 2556 ms. Trace:
I20260806 04:08:16.064751 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:16.762475 4995 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.004s sys 0.008s
W20260806 04:08:16.874580 4995 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.882s user 0.005s sys 0.008s
I20260806 04:08:16.825704 8692 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
I20260806 04:08:16.428805 5037 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
W20260806 04:08:16.875124 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 882618us
W20260806 04:08:16.876296 5037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1749) took 1700 ms (client timeout 1638 ms). Trace:
W20260806 04:08:16.876845 5037 rpcz_store.cc:269] 0806 04:08:15.175396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.672837 (+497441us) service_pool.cc:227] Handling call
0806 04:08:16.875916 (+1203079us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:16.775313 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 215444us
W20260806 04:08:16.777230 5812 leader_election.cc:334] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.924s (SENT)
I20260806 04:08:16.879117 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:16.881625 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 104711us
I20260806 04:08:16.778717 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:16.770298 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2083) took 2554 ms. Trace:
W20260806 04:08:16.882925 4927 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1560) took 1285 ms (client timeout 1653 ms). Trace:
W20260806 04:08:16.622068 8200 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.007s sys 0.024s
W20260806 04:08:16.883764 8200 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s user 0.008s sys 0.024s
I20260806 04:08:16.770273 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:16.814949 4805 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.944s user 0.002s sys 0.016s
I20260806 04:08:16.826303 8954 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:16.829635 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1713) took 3906 ms. Trace:
I20260806 04:08:16.841291 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1424) took 1938 ms. Trace:
I20260806 04:08:16.883136 4809 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:16.830665 8944 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:08:16.872730 8959 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s user 0.004s sys 0.020s
I20260806 04:08:16.873399 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:16.873801 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 125243us
W20260806 04:08:16.493827 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1715) took 2272 ms (client timeout 2695 ms). Trace:
W20260806 04:08:16.895144 5035 rpcz_store.cc:269] 0806 04:08:14.221879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.394583 (+2172704us) service_pool.cc:227] Handling call
0806 04:08:16.493761 (+ 99178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.896694 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:16.883111 4878 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.373s user 0.024s sys 0.025s
W20260806 04:08:16.788663 5814 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.055s sys 0.143s
I20260806 04:08:16.846378 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:16.882562 5812 leader_election.cc:334] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.972s (SENT)
I20260806 04:08:16.885051 4810 rpcz_store.cc:276] 0806 04:08:14.902373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.902567 (+ 194us) service_pool.cc:227] Handling call
0806 04:08:16.839566 (+1936999us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:16.840037 (+ 471us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:16.840545 (+ 508us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:16.840618 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:16.840695 (+ 77us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:16.840747 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:16.840817 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:16.840986 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23883648}
I20260806 04:08:16.845103 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:16.874837 9041 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 04:08:16.884524 4927 rpcz_store.cc:269] 0806 04:08:15.588664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.589755 (+ 1091us) service_pool.cc:227] Handling call
0806 04:08:16.874173 (+1284418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.875042 4928 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
W20260806 04:08:16.891063 5859 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.024s sys 0.010s
I20260806 04:08:16.893070 8954 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:16.893604 8708 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:16.908497 5859 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.676s user 0.024s sys 0.011s
I20260806 04:08:16.894117 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:16.897746 5035 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
W20260806 04:08:16.911729 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.002s sys 0.006s
W20260806 04:08:16.912226 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.887s user 0.002s sys 0.006s
I20260806 04:08:16.916455 9062 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:16.916790 8391 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:16.917397 9062 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:16.906934 8959 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s user 0.004s sys 0.020s
I20260806 04:08:16.919518 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:16.919703 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:16.920367 8959 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:16.921083 8959 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:16.923285 4817 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:16.923409 8944 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:16.923985 4817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1426) took 1997 ms (client timeout 2013 ms). Trace:
W20260806 04:08:16.924472 4817 rpcz_store.cc:269] 0806 04:08:14.926375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.947070 (+ 20695us) service_pool.cc:227] Handling call
0806 04:08:16.769705 (+1822635us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:16.923892 (+154187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18519296}
I20260806 04:08:16.917493 8391 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:16.917917 5032 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:16.922528 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:16.835518 8942 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:16.884742 4923 rpcz_store.cc:276] 0806 04:08:12.923475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.923640 (+ 165us) service_pool.cc:227] Handling call
0806 04:08:12.924084 (+ 444us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:14.588372 (+1664288us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:14.588401 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:14.588436 (+ 35us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:14.588475 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:16.822420 (+2233945us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540018fbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:16.829082 (+ 6662us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:16.829364 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9285,"raft.queue_time_us":214544,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":39291,"spinlock_wait_cycles":3251632512}
W20260806 04:08:16.928366 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 316) took 1532 ms (client timeout 1887 ms). Trace:
I20260806 04:08:16.931977 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:16.933516 5035 rpcz_store.cc:269] 0806 04:08:15.378383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.896210 (+1517827us) service_pool.cc:227] Handling call
0806 04:08:16.910441 (+ 14231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.883298 5036 rpcz_store.cc:276] 0806 04:08:14.215865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.216017 (+ 152us) service_pool.cc:227] Handling call
0806 04:08:14.291424 (+ 75407us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:14.292135 (+ 711us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:14.292174 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:14.292892 (+ 718us) log.cc:842] Serialized 52 byte log entry
0806 04:08:14.403679 (+110787us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:15.124421 (+720742us) log.cc:842] Serialized 14 byte log entry
0806 04:08:15.125170 (+ 749us) log.cc:842] Serialized 14 byte log entry
0806 04:08:15.125360 (+ 190us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:15.125458 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:16.769422 (+1643964us) raft_consensus.cc:1671] finished
0806 04:08:16.769475 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:16.769947 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12645,"prepare.queue_time_us":163091,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":4464,"raft.queue_time_us":55907,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":33125,"spinlock_wait_cycles":31491456,"thread_start_us":786058,"threads_started":2,"wal-append.queue_time_us":1069866}
I20260806 04:08:16.937875 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:16.938979 9045 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:16.939543 8783 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 6.540s user 0.038s sys 0.078s
I20260806 04:08:16.938983 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:16.931725 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1752) took 1750 ms (client timeout 1976 ms). Trace:
W20260806 04:08:16.942407 5032 rpcz_store.cc:269] 0806 04:08:15.176280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.893473 (+1717193us) service_pool.cc:227] Handling call
0806 04:08:16.926933 (+ 33460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.943708 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:16.943969 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2091) took 2726 ms (client timeout 2052 ms). Trace:
I20260806 04:08:16.944108 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:16.944636 5032 raft_consensus.cc:2354] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
W20260806 04:08:16.944410 5034 rpcz_store.cc:269] 0806 04:08:14.217920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.153526 (+935606us) service_pool.cc:227] Handling call
0806 04:08:16.943561 (+1790035us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400167954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:16.943785 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1129705344}
I20260806 04:08:16.945742 4920 raft_consensus.cc:2354] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:16.945848 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:16.946713 4816 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:16.948024 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1566) took 1076 ms. Trace:
I20260806 04:08:16.948408 4931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 409) took 7233 ms. Trace:
W20260806 04:08:16.948642 9076 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.102s user 0.002s sys 0.000s
I20260806 04:08:16.933245 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:16.959899 5037 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1746) took 1895 ms (client timeout 2085 ms). Trace:
W20260806 04:08:16.960356 5037 rpcz_store.cc:269] 0806 04:08:15.064810 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.878415 (+1813605us) service_pool.cc:227] Handling call
0806 04:08:16.959795 (+ 81380us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.961670 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:16.962741 4768 leader_election.cc:334] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.854s (SENT)
I20260806 04:08:16.963785 8815 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:16.963990 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:16.964933 4768 leader_election.cc:334] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.883s (SENT)
I20260806 04:08:16.965934 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:16.966321 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:16.966526 4928 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:16.924464 8944 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:16.968215 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:16.969097 5863 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:16.949882 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:16.970719 4923 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:16.952545 4814 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
W20260806 04:08:16.954177 8878 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 4.556s user 0.009s sys 0.034s
I20260806 04:08:16.940231 9045 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:16.834594 8534 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:16.945240 5032 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1754) took 1768 ms (client timeout 2247 ms). Trace:
W20260806 04:08:16.998874 5032 rpcz_store.cc:269] 0806 04:08:15.176888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.943227 (+1766339us) service_pool.cc:227] Handling call
0806 04:08:16.945160 (+ 1933us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.945252 5035 raft_consensus.cc:2396] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 04:08:16.964866 5037 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:08:16.938522 5036 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:16.837558 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:16.972167 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:16.958750 4925 rpcz_store.cc:276] 0806 04:08:15.871345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.933525 (+ 62180us) service_pool.cc:227] Handling call
0806 04:08:16.947799 (+1014274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:16.909271 8692 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:17.000145 5036 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1750) took 1824 ms (client timeout 2220 ms). Trace:
W20260806 04:08:17.005192 5036 rpcz_store.cc:269] 0806 04:08:15.175702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.937453 (+1761751us) service_pool.cc:227] Handling call
0806 04:08:17.000074 (+ 62621us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.006280 5034 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:17.006469 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:17.007012 5034 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1755) took 1829 ms (client timeout 2249 ms). Trace:
I20260806 04:08:16.959124 4931 rpcz_store.cc:276] 0806 04:08:09.715088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:09.715551 (+ 463us) service_pool.cc:227] Handling call
0806 04:08:09.899375 (+183824us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:09.992296 (+ 92921us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:09.992348 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:10.008312 (+ 15964us) log.cc:842] Serialized 52 byte log entry
0806 04:08:10.121386 (+113074us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:10.384015 (+262629us) log.cc:842] Serialized 14 byte log entry
0806 04:08:10.403505 (+ 19490us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:10.403610 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:13.153718 (+2750108us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b18003208c0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69fff96 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511
0806 04:08:16.850101 (+3696383us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:16.947888 (+ 97787us) raft_consensus.cc:1671] finished
0806 04:08:16.947914 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:16.948165 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104659,"prepare.queue_time_us":390700,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":118616,"raft.run_cpu_time_us":493,"raft.run_wall_time_us":31767,"spinlock_wait_cycles":22545024,"thread_start_us":291950,"threads_started":2,"wal-append.queue_time_us":219391}
I20260806 04:08:17.007694 5032 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:17.008807 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.009369 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:16.865965 4930 rpcz_store.cc:276] 0806 04:08:14.202454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.202606 (+ 152us) service_pool.cc:227] Handling call
0806 04:08:14.203017 (+ 411us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:14.450045 (+247028us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:14.450094 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:14.450706 (+ 612us) log.cc:842] Serialized 26 byte log entry
0806 04:08:14.490447 (+ 39741us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:14.491083 (+ 636us) log.cc:842] Serialized 14 byte log entry
0806 04:08:14.491312 (+ 229us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:14.491416 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:16.757962 (+2266546us) raft_consensus.cc:1671] finished
0806 04:08:16.758014 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:16.758326 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5763,"prepare.queue_time_us":387,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":118057,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":44320,"thread_start_us":274152,"threads_started":2,"wal-append.queue_time_us":973948}
W20260806 04:08:17.006539 4768 leader_election.cc:334] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.920s (SENT)
I20260806 04:08:17.009792 4931 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 9: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
W20260806 04:08:17.007556 5034 rpcz_store.cc:269] 0806 04:08:15.177149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.945403 (+1768254us) service_pool.cc:227] Handling call
0806 04:08:17.006940 (+ 61537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.021253 4768 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:17.019634 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1420) took 2452 ms. Trace:
I20260806 04:08:17.016089 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.010399 5032 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:17.015583 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.015446 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1753) took 1838 ms (client timeout 2200 ms). Trace:
W20260806 04:08:17.024503 5035 rpcz_store.cc:269] 0806 04:08:15.176605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.943715 (+1767110us) service_pool.cc:227] Handling call
0806 04:08:17.015379 (+ 71664us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:17.010438 8935 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.014s sys 0.004s
I20260806 04:08:17.020249 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.022862 4814 rpcz_store.cc:276] 0806 04:08:14.552087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.560857 (+ 8770us) service_pool.cc:227] Handling call
0806 04:08:14.561190 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:16.952130 (+2390940us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540028fb20. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d98ea730 00007f23d995c300 00007f23d995d371 00007f23d9959edf 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc
0806 04:08:16.988396 (+ 36266us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.004631 (+ 16235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1396104064}
I20260806 04:08:17.043121 4923 raft_consensus.cc:2354] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8.
I20260806 04:08:17.044337 8692 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:08:17.047168 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:17.051857 5026 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.486s user 0.009s sys 0.012s
I20260806 04:08:17.052417 5026 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
W20260806 04:08:17.053009 5026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1738) took 2631 ms (client timeout 1898 ms). Trace:
W20260806 04:08:17.053315 5026 rpcz_store.cc:269] 0806 04:08:14.421882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.160115 (+738233us) service_pool.cc:227] Handling call
0806 04:08:17.052925 (+1892810us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.055583 8944 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.043200 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:17.044384 5038 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.275s user 0.010s sys 0.009s
W20260806 04:08:17.032816 8935 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s user 0.015s sys 0.005s
W20260806 04:08:17.064478 5853 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 695) took 3070 ms (client timeout 1777 ms). Trace:
I20260806 04:08:17.071780 4931 raft_consensus.cc:3032] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:08:17.071990 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:17.071837 4808 consensus_meta.cc:287] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.922s user 0.008s sys 0.016s
I20260806 04:08:17.072609 4931 raft_consensus.cc:729] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:17.071930 5038 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
W20260806 04:08:17.083132 5038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1751) took 1907 ms (client timeout 1606 ms). Trace:
W20260806 04:08:17.083607 5038 rpcz_store.cc:269] 0806 04:08:15.176005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.671496 (+495491us) service_pool.cc:227] Handling call
0806 04:08:17.074279 (+1402783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.074584 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2087) took 2848 ms. Trace:
I20260806 04:08:17.086378 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.071789 5853 rpcz_store.cc:269] 0806 04:08:13.993977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:13.994895 (+ 918us) service_pool.cc:227] Handling call
0806 04:08:16.837238 (+2842343us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fac6c 00007f23d69fabd6 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:17.064351 (+227113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66755456}
I20260806 04:08:17.086726 5026 rpcz_store.cc:276] 0806 04:08:14.217055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.053991 (+2836936us) service_pool.cc:227] Handling call
0806 04:08:17.054432 (+ 441us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.059650 (+ 5218us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:17.059690 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:17.065375 (+ 5685us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:17.065436 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.065514 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.065651 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.098284 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:17.099588 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.113361 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 106117us
I20260806 04:08:17.120460 9007 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.123787 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 696) took 1254 ms. Trace:
W20260806 04:08:17.125327 8785 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 5.883s user 0.015s sys 0.058s
W20260806 04:08:17.125773 8785 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 5.981s user 0.016s sys 0.058s
I20260806 04:08:17.126593 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.127174 5026 raft_consensus.cc:1203] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-5.4] Dedup: 3.3->[5.4-5.4]
I20260806 04:08:17.127737 5026 raft_consensus.cc:1226] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260806 04:08:17.128343 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2090) took 2910 ms. Trace:
I20260806 04:08:17.128662 5026 rpcz_store.cc:276] 0806 04:08:14.217661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.097079 (+2879418us) service_pool.cc:227] Handling call
0806 04:08:17.126827 (+ 29748us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:17.128066 (+ 1239us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.128273 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.130221 5026 raft_consensus.cc:1226] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:17.131299 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:17.134939 4882 leader_election.cc:334] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.511s (SENT)
I20260806 04:08:17.135528 4882 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:17.136713 8768 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:08:17.139523 9053 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:17.140290 9053 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.141088 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2092) took 2922 ms. Trace:
I20260806 04:08:17.141433 5026 rpcz_store.cc:276] 0806 04:08:14.218194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.129456 (+2911262us) service_pool.cc:227] Handling call
0806 04:08:17.129899 (+ 443us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.140825 (+ 10926us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.141032 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.142997 9053 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.146248 8954 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:17.146994 8959 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.124221 5859 rpcz_store.cc:276] 0806 04:08:15.869555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.869727 (+ 172us) service_pool.cc:227] Handling call
0806 04:08:16.005849 (+136122us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:16.006077 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:08:16.937699 (+931622us) log.cc:842] Serialized 14 byte log entry
0806 04:08:17.068977 (+131278us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:17.086353 (+ 17376us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:17.108111 (+ 21758us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.114285 (+ 6174us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.123704 (+ 9419us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":948,"prepare.queue_time_us":620698,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":33308,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":23522,"thread_start_us":764653,"threads_started":2}
I20260806 04:08:17.114436 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.111639 4808 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:17.152470 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1427) took 2204 ms. Trace:
I20260806 04:08:17.152673 9081 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:17.152804 4804 rpcz_store.cc:276] 0806 04:08:14.947630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.947818 (+ 188us) service_pool.cc:227] Handling call
0806 04:08:16.970356 (+2022538us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:16.970781 (+ 425us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:16.971203 (+ 422us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:08:16.971727 (+ 524us) log.cc:842] Serialized 14 byte log entry
0806 04:08:17.151999 (+180272us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:17.152056 (+ 57us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:17.152142 (+ 86us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.152220 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.152337 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34284,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":4409,"raft.queue_time_us":117,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"spinlock_wait_cycles":18926976,"thread_start_us":179781,"threads_started":1}
I20260806 04:08:17.153663 9081 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.154856 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.163451 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.162242 4924 raft_consensus.cc:2454] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:17.172302 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.178352 8534 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.178562 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 306) took 4372 ms. Trace:
I20260806 04:08:17.179165 5029 rpcz_store.cc:276] 0806 04:08:12.805892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.832359 (+ 26467us) service_pool.cc:227] Handling call
0806 04:08:12.899348 (+ 66989us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:12.912726 (+ 13378us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.178070 (+4265344us) spinlock_profiling.cc:243] Waited 66.6 ms on lock 0x7b54002440d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.178310 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":153222016}
I20260806 04:08:17.181716 5029 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:08:17.183312 9062 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.137995 9060 consensus_queue.cc:1046] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:17.186336 5037 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:08:17.188022 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1756) took 2010 ms. Trace:
W20260806 04:08:17.175612 4808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1431) took 2212 ms (client timeout 2017 ms). Trace:
I20260806 04:08:17.188503 5037 rpcz_store.cc:276] 0806 04:08:15.177425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.963353 (+1785928us) service_pool.cc:227] Handling call
0806 04:08:17.187951 (+224598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.188864 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.189311 4816 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:17.182488 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1741) took 2689 ms. Trace:
I20260806 04:08:17.190061 5029 rpcz_store.cc:276] 0806 04:08:14.493652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.180494 (+2686842us) service_pool.cc:227] Handling call
0806 04:08:17.181090 (+ 596us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.182161 (+ 1071us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.182398 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.118010 8944 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:17.185062 4882 leader_election.cc:334] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.524s (SENT)
W20260806 04:08:17.188621 4808 rpcz_store.cc:269] 0806 04:08:14.956621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.131143 (+174522us) service_pool.cc:227] Handling call
0806 04:08:17.164845 (+2033702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.195039 5038 raft_consensus.cc:1226] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 1. Current term is 2. Ops: []
I20260806 04:08:17.196110 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.121248 9007 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.200790 4927 raft_consensus.cc:1261] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:17.199870 4814 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:17.208576 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.209350 8977 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:08:17.210788 4995 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:17.168533 8692 consensus_queue.cc:1046] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 04:08:17.212185 8873 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:17.212637 8873 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:17.212994 8873 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:17.214704 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.216609 4926 raft_consensus.cc:2396] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:17.218184 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 439) took 2681 ms. Trace:
I20260806 04:08:17.218953 8936 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.219525 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.089658 8942 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.241214 8750 raft_consensus.cc:3032] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:17.241084 8936 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.241914 8750 raft_consensus.cc:729] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:17.248309 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.247025 4920 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:17.250384 4931 consensus_queue.cc:260] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.251302 5034 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:08:17.252054 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1731) took 2970 ms. Trace:
I20260806 04:08:17.252261 4931 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:08:17.252465 5034 rpcz_store.cc:276] 0806 04:08:14.281426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.188455 (+2907029us) service_pool.cc:227] Handling call
0806 04:08:17.250769 (+ 62314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.251726 (+ 957us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.251951 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.255285 9045 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.195319 9047 consensus_queue.cc:1046] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260806 04:08:17.257208 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.257475 9047 raft_consensus.cc:3032] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:17.257256 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.258018 9047 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:17.258826 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.259640 4808 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:17.260042 9047 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.258672 8952 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:17.261803 8750 consensus_queue.cc:260] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.262244 8952 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.263069 8750 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:17.264439 9007 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.265113 4924 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:17.265309 4809 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:17.265919 4817 raft_consensus.cc:2396] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 04:08:17.266196 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 99166.7us
I20260806 04:08:17.266471 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.270558 4920 raft_consensus.cc:1261] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
W20260806 04:08:17.273043 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 270720us
I20260806 04:08:17.273123 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.174924 8401 consensus_queue.cc:1035] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.094s
I20260806 04:08:17.274403 4930 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260806 04:08:17.274791 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.276777 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.273783 4808 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:17.276414 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.265456 8952 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:17.283620 8900 log.cc:924] Time spent T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 4.570s user 0.013s sys 0.047s
I20260806 04:08:17.281075 4817 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:17.288443 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.290997 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.294942 4928 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:17.218626 4805 rpcz_store.cc:276] 0806 04:08:14.537002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.554917 (+ 17915us) service_pool.cc:227] Handling call
0806 04:08:14.555261 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:15.701833 (+1146572us) raft_consensus.cc:1520] Early marking committed up to index 9
0806 04:08:16.892137 (+1190304us) log.cc:842] Serialized 14 byte log entry
0806 04:08:17.031435 (+139298us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:17.130927 (+ 99492us) log.cc:842] Serialized 26 byte log entry
0806 04:08:17.162367 (+ 31440us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:08:17.162412 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.168640 (+ 6228us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:17.217939 (+ 49299us) raft_consensus.cc:1671] finished
0806 04:08:17.217960 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.218121 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53129,"prepare.queue_time_us":742893,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":600755,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":20192,"thread_start_us":2202099,"threads_started":3,"wal-append.queue_time_us":216495}
I20260806 04:08:17.296862 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:17.298621 4928 raft_consensus.cc:1261] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:08:17.298969 4816 raft_consensus.cc:2362] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5. Re-sending same reply.
W20260806 04:08:17.300356 4882 leader_election.cc:334] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.548s (SENT)
I20260806 04:08:17.300630 4928 raft_consensus.cc:1226] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:17.300837 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.301928 4928 raft_consensus.cc:1226] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:17.302112 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.303332 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.303347 4928 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:17.304759 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.305325 4928 raft_consensus.cc:1261] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:08:17.305841 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.305954 8980 raft_consensus.cc:3032] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Stepping down as leader of term 5
W20260806 04:08:17.302438 4882 leader_election.cc:334] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.615s (SENT)
I20260806 04:08:17.308597 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.298419 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.306411 8980 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:17.311504 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2086) took 3082 ms. Trace:
W20260806 04:08:17.312690 4768 leader_election.cc:334] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.926s (SENT)
I20260806 04:08:17.287848 8942 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.320108 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.321363 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.320538 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:17.321745 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:17.322925 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.324870 5036 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:08:17.324558 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.326983 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.312952 5032 rpcz_store.cc:276] 0806 04:08:14.216779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.037461 (+2820682us) service_pool.cc:227] Handling call
0806 04:08:17.182495 (+145034us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:17.183165 (+ 670us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:17.183195 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:17.187035 (+ 3840us) log.cc:842] Serialized 52 byte log entry
0806 04:08:17.268500 (+ 81465us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:17.268557 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.268649 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:17.296364 (+ 27715us) raft_consensus.cc:1671] finished
0806 04:08:17.296392 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.298748 (+ 2356us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11976,"raft.queue_time_us":904,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":12430,"thread_start_us":57235,"threads_started":1,"wal-append.queue_time_us":79588}
I20260806 04:08:17.329116 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 315) took 1947 ms. Trace:
I20260806 04:08:17.329417 5036 rpcz_store.cc:276] 0806 04:08:15.377994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.020912 (+1642918us) service_pool.cc:227] Handling call
0806 04:08:17.307788 (+286876us) spinlock_profiling.cc:243] Waited 7.09 ms on lock 0x7b5800004898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:17.325468 (+ 17680us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16302336}
W20260806 04:08:17.332635 5854 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.922s user 0.009s sys 0.011s
I20260806 04:08:17.333316 5854 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:17.334009 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2088) took 3116 ms. Trace:
I20260806 04:08:17.334407 5038 rpcz_store.cc:276] 0806 04:08:14.217316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.088858 (+2871542us) service_pool.cc:227] Handling call
0806 04:08:17.187944 (+ 99086us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.195486 (+ 7542us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.333747 (+138261us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b5400320ad0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.333922 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171062656}
I20260806 04:08:17.338253 5038 raft_consensus.cc:1261] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:17.325289 4809 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
W20260806 04:08:17.341732 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 997e4d9ff3ca465199a888e8af23ed7b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:17.342217 8614 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:17.346717 8780 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.347450 8780 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.318538 8952 consensus_queue.cc:1046] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260806 04:08:17.338400 8463 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.349653 8936 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.431017 5037 raft_consensus.cc:1261] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 04:08:17.352602 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:17.419575 8899 consensus_queue.cc:1035] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.002s
W20260806 04:08:17.436165 4992 leader_election.cc:334] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.907s (SENT)
I20260806 04:08:17.435480 5035 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:17.436851 4992 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:17.437770 8862 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:17.437372 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2094) took 1126 ms. Trace:
I20260806 04:08:17.439065 5035 rpcz_store.cc:276] 0806 04:08:16.310825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.352233 (+1041408us) service_pool.cc:227] Handling call
0806 04:08:17.437315 (+ 85082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.405865 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.440459 4925 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:17.441435 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:17.442256 4925 raft_consensus.cc:2362] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4. Re-sending same reply.
I20260806 04:08:17.442842 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 436) took 2928 ms. Trace:
I20260806 04:08:17.443216 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:17.444609 8463 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:17.443418 4818 rpcz_store.cc:276] 0806 04:08:14.514460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.530865 (+ 16405us) service_pool.cc:227] Handling call
0806 04:08:14.586313 (+ 55448us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:15.171973 (+585660us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:08:16.847461 (+1675488us) log.cc:842] Serialized 14 byte log entry
0806 04:08:17.071221 (+223760us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:17.071324 (+ 103us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:17.090152 (+ 18828us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.442368 (+352216us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b540039a8d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:17.442415 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.442614 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2794,"prepare.queue_time_us":1203940,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":25089,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":10008,"spinlock_wait_cycles":1346300288,"thread_start_us":1757782,"threads_started":2,"wal-append.queue_time_us":196762}
I20260806 04:08:17.447484 8962 raft_consensus.cc:3032] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:17.447925 8962 raft_consensus.cc:729] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:17.449187 8962 consensus_queue.cc:260] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.450536 8962 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:17.451493 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.452571 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.453966 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.334625 5854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 643) took 1735 ms. Trace:
I20260806 04:08:17.349673 8780 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.417117 8732 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:17.421499 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:17.363196 8979 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 3.022s user 0.015s sys 0.037s
I20260806 04:08:17.404114 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2098) took 1024 ms. Trace:
I20260806 04:08:17.468153 5038 rpcz_store.cc:276] 0806 04:08:16.379746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.336041 (+956295us) service_pool.cc:227] Handling call
0806 04:08:17.337821 (+ 1780us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.338617 (+ 796us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.404047 (+ 65430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.366312 4816 raft_consensus.cc:2396] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 04:08:17.409825 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 107479us
I20260806 04:08:17.346406 8873 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.346446 8391 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:17.410868 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 100118us
W20260806 04:08:17.415817 8855 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 5.774s user 0.010s sys 0.048s
I20260806 04:08:17.358247 4930 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:17.384825 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.424507 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.379499 4926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1562) took 1785 ms (client timeout 2178 ms). Trace:
I20260806 04:08:17.386739 9062 raft_consensus.cc:3032] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:17.397084 8756 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.425982 8921 raft_consensus.cc:3032] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:17.425552 5029 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:17.393440 9081 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.428484 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 641) took 1851 ms. Trace:
W20260806 04:08:17.430121 5028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1736) took 3009 ms (client timeout 1536 ms). Trace:
I20260806 04:08:17.479748 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:17.479952 5028 rpcz_store.cc:269] 0806 04:08:14.421038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:14.473891 (+ 52853us) service_pool.cc:227] Handling call
0806 04:08:17.429572 (+2955681us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b540040e7d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.429799 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1926278400}
I20260806 04:08:17.480454 4930 raft_consensus.cc:2362] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7. Re-sending same reply.
I20260806 04:08:17.421360 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.479481 5857 rpcz_store.cc:276] 0806 04:08:15.576951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.633636 (+ 56685us) service_pool.cc:227] Handling call
0806 04:08:17.427350 (+1793714us) spinlock_profiling.cc:243] Waited 98.9 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fabce 000000000034aea9 00007f23de826a30 00007f23de8ec7b4 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:17.427583 (+ 233us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.427881 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:17.427920 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:17.427968 (+ 48us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:17.428007 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.428088 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.428244 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227522432}
I20260806 04:08:17.481431 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:17.481828 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.482144 4930 raft_consensus.cc:2362] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7. Re-sending same reply.
I20260806 04:08:17.483098 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.483989 8873 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.484334 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.475510 9062 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:17.485246 4923 raft_consensus.cc:2379] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:17.485787 9062 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.432196 5863 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:17.368880 4804 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:17.370137 9089 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20260806 04:08:17.353474 5856 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 635) took 3409 ms (client timeout 2013 ms). Trace:
I20260806 04:08:17.405592 5036 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 04:08:17.446448 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.400066 5034 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:17.492951 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2095) took 1152 ms. Trace:
I20260806 04:08:17.493286 5034 rpcz_store.cc:276] 0806 04:08:16.350502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.340129 (+989627us) service_pool.cc:227] Handling call
0806 04:08:17.399803 (+ 59674us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.492698 (+ 92895us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.492895 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.444007 4925 raft_consensus.cc:2354] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:17.454707 4814 raft_consensus.cc:2379] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 7: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:17.463177 4931 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 2.
I20260806 04:08:17.463665 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2093) took 3245 ms. Trace:
W20260806 04:08:17.473451 4926 rpcz_store.cc:269] 0806 04:08:15.594043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.595037 (+ 994us) service_pool.cc:227] Handling call
0806 04:08:17.379174 (+1784137us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54001b76d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.379416 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3725075840}
I20260806 04:08:17.351663 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.495276 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.495680 4928 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:17.496102 4813 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:17.496984 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.497412 4928 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 04:08:17.478973 4768 leader_election.cc:334] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.935s (SENT)
I20260806 04:08:17.475445 8921 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:17.431017 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.476065 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.505817 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.475365 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 53980.4us
I20260806 04:08:17.368418 8975 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:17.495484 4818 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:17.479130 8952 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 04:08:17.472313 4882 leader_election.cc:334] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.665s (SENT)
I20260806 04:08:17.503981 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2097) took 1124 ms. Trace:
I20260806 04:08:17.518620 5036 rpcz_store.cc:276] 0806 04:08:16.379389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.393130 (+1013741us) service_pool.cc:227] Handling call
0806 04:08:17.405207 (+ 12077us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:17.503556 (+ 98349us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.503910 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.520447 4768 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:17.519824 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.520129 8952 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:17.523594 9028 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260806 04:08:17.523931 8952 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:17.524459 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:17.526611 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 54342.4us
I20260806 04:08:17.466284 5854 rpcz_store.cc:276] 0806 04:08:15.599413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.621626 (+ 22213us) service_pool.cc:227] Handling call
0806 04:08:17.334403 (+1712777us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.527014 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.526783 4809 raft_consensus.cc:1261] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:08:17.494379 5026 rpcz_store.cc:276] 0806 04:08:14.218461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.142222 (+2923761us) service_pool.cc:227] Handling call
0806 04:08:17.142778 (+ 556us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:17.176760 (+ 33982us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:17.176791 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:17.177397 (+ 606us) log.cc:842] Serialized 52 byte log entry
0806 04:08:17.271433 (+ 94036us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:17.314700 (+ 43267us) log.cc:842] Serialized 14 byte log entry
0806 04:08:17.316152 (+ 1452us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.316225 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:17.458642 (+142417us) raft_consensus.cc:1671] finished
0806 04:08:17.458682 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.463581 (+ 4899us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25048,"prepare.queue_time_us":41040,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":5779,"raft.queue_time_us":35417,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":3854,"thread_start_us":93542,"threads_started":1,"wal-append.queue_time_us":147370}
I20260806 04:08:17.531560 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.383189 8980 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.545590 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:17.539752 9062 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:17.544775 8693 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.004s
I20260806 04:08:17.554265 8945 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:08:17.554728 8730 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260806 04:08:17.555904 8950 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:17.530124 8756 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:17.571143 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:17.572958 4995 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:17.503003 5856 rpcz_store.cc:269] 0806 04:08:13.943545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:13.945905 (+ 2360us) service_pool.cc:227] Handling call
0806 04:08:17.353079 (+3407174us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400770954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.353273 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":533034752}
I20260806 04:08:17.576133 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.579623 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:17.587338 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.589886 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.583583 8756 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.593639 8756 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:17.564173 8950 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:17.590761 4876 connection.cc:232] RPC call timeout handler was delayed by -1.3030151280108839s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
I20260806 04:08:17.594986 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.596227 4876 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:17.596361 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.597644 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:17.597774 4882 leader_election.cc:334] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.729s (SENT)
W20260806 04:08:17.599145 4882 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.746s (SENT)
I20260806 04:08:17.599620 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.591063 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:17.600951 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:17.591351 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.601742 4928 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:17.566455 8980 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:17.594410 8950 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:17.599848 8995 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:17.591179 9002 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:17.605189 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 129885us
I20260806 04:08:17.606848 5816 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:17.606925 5860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 345) took 4788 ms. Trace:
I20260806 04:08:17.610847 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.614415 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 62443.1us
I20260806 04:08:17.620791 4928 raft_consensus.cc:1226] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:17.620668 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1748) took 2445 ms. Trace:
I20260806 04:08:17.622437 5037 rpcz_store.cc:276] 0806 04:08:15.174965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.203544 (+2028579us) service_pool.cc:227] Handling call
0806 04:08:17.430583 (+227039us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.434083 (+ 3500us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.620525 (+186442us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540040e7d0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.620622 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":403678080}
I20260806 04:08:17.625788 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:17.628412 4818 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 9.
I20260806 04:08:17.629912 4818 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
W20260806 04:08:17.633567 4882 leader_election.cc:334] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.868s (SENT)
I20260806 04:08:17.600992 8641 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.606889 8180 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:17.647342 4817 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: [7.9-8.10]
I20260806 04:08:17.644124 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:17.650532 4768 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.936s (SENT)
I20260806 04:08:17.654664 4930 raft_consensus.cc:1226] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:17.671722 4926 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 9: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:17.672014 4816 raft_consensus.cc:3032] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:17.677379 4816 raft_consensus.cc:729] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:17.677786 4813 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:17.644625 8862 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:17.655454 9047 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:17.695945 8693 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 04:08:17.702283 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 76599.1us
I20260806 04:08:17.611541 5860 rpcz_store.cc:276] 0806 04:08:12.818039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:12.818196 (+ 157us) service_pool.cc:227] Handling call
0806 04:08:13.962305 (+1144109us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:13.962516 (+ 211us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:13.962965 (+ 449us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:13.963007 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:13.963552 (+ 545us) log.cc:842] Serialized 26 byte log entry
0806 04:08:14.084241 (+120689us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:14.149266 (+ 65025us) log.cc:842] Serialized 14 byte log entry
0806 04:08:14.149504 (+ 238us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:14.149573 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:17.315046 (+3165473us) spinlock_profiling.cc:243] Waited 5.03 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:17.605349 (+290303us) raft_consensus.cc:1671] finished
0806 04:08:17.605385 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:17.606727 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15700,"prepare.queue_time_us":208460,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":14432,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":35712,"spinlock_wait_cycles":14478592,"thread_start_us":117762,"threads_started":2,"wal-append.queue_time_us":203853}
W20260806 04:08:17.708200 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 91988.4us
I20260806 04:08:17.706354 4920 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:17.701620 4816 consensus_queue.cc:260] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.722515 8456 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.723317 8456 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.724877 5030 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:17.727136 5859 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:17.683512 8641 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:17.732324 9106 log.cc:924] Time spent T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260806 04:08:17.701375 8180 raft_consensus.cc:686] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:17.736413 4813 raft_consensus.cc:3032] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:17.739499 4813 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:08:17.744755 4882 leader_election.cc:334] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.907s (SENT)
I20260806 04:08:17.741905 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:17.741181 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:17.730388 8641 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.708187 8862 raft_consensus.cc:686] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:17.760748 4882 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:17.761133 4926 raft_consensus.cc:2379] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:17.768633 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.769956 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.769105 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:17.774389 4816 raft_consensus.cc:3037] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:17.688827 9002 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:17.777592 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 43585.1us
I20260806 04:08:17.779807 4818 raft_consensus.cc:3032] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:17.780202 4818 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:17.783977 8180 consensus_queue.cc:237] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.687104 8995 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:17.795078 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 175110us
W20260806 04:08:17.795902 4768 leader_election.cc:334] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.003s (SENT)
W20260806 04:08:17.796034 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 51337.7us
I20260806 04:08:17.796303 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:17.796586 8995 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:17.797566 4768 leader_election.cc:334] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.134s (SENT)
I20260806 04:08:17.798025 4768 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:17.797721 8862 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.804844 8641 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:17.784539 9054 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:17.813009 5029 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:08:17.781996 4813 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:17.815573 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 317) took 2436 ms. Trace:
I20260806 04:08:17.815917 5029 rpcz_store.cc:276] 0806 04:08:15.378737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.373611 (+1994874us) service_pool.cc:227] Handling call
0806 04:08:17.425407 (+ 51796us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.815317 (+389910us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.815522 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:17.852661 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1572) took 1062 ms. Trace:
I20260806 04:08:17.853156 4924 rpcz_store.cc:276] 0806 04:08:16.756827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.174807 (+417980us) service_pool.cc:227] Handling call
0806 04:08:17.176560 (+ 1753us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.265542 (+ 88982us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:17.818871 (+553329us) spinlock_profiling.cc:243] Waited 88.3 ms on lock 0x7b54003aff54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:17.819094 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":203203968}
I20260806 04:08:17.859520 4808 raft_consensus.cc:2396] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 04:08:17.860644 4816 raft_consensus.cc:1261] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:17.859520 4804 raft_consensus.cc:2396] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:17.863138 4813 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:17.861615 4818 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:17.862708 4919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1557) took 2275 ms (client timeout 2391 ms). Trace:
I20260806 04:08:17.748123 8952 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:17.866623 4919 rpcz_store.cc:269] 0806 04:08:15.580950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:15.582562 (+ 1612us) service_pool.cc:227] Handling call
0806 04:08:17.819341 (+2236779us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540004fbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:17.856051 (+ 36710us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2695540608}
I20260806 04:08:17.815003 9054 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:17.694496 8921 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:17.875020 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 186460us
I20260806 04:08:17.859315 8945 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:17.884120 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 71187.1us
I20260806 04:08:17.884289 4992 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:17.884719 4804 raft_consensus.cc:1226] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 04:08:17.891656 4818 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:17.806548 8950 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.872423 9054 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:17.899299 4804 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:17.909781 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 112242us
I20260806 04:08:17.901469 5853 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:17.914512 8984 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 04:08:17.915143 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 44190.7us
I20260806 04:08:17.897774 8750 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.868872 9094 consensus_queue.cc:1046] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 04:08:17.902150 8921 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:17.950570 4992 leader_election.cc:398] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8.
I20260806 04:08:17.950860 8631 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260806 04:08:17.951045 8750 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:17.953791 4808 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:17.956822 4992 leader_election.cc:401] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:17.914156 8981 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:17.961086 5863 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:17.951979 8693 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
W20260806 04:08:17.962853 4768 leader_election.cc:334] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.614s (SENT)
I20260806 04:08:17.975395 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:17.983565 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 82413.8us
I20260806 04:08:17.961948 8730 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:17.991940 5812 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:17.983203 8997 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:18.002781 9060 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:18.002194 8730 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:18.029817 8873 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:18.055058 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 63214.4us
W20260806 04:08:18.060379 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 97584.2us
I20260806 04:08:18.062398 5812 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.065826 5812 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.071734 9062 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:18.072002 8757 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:18.071882 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 121549us
I20260806 04:08:18.071573 8532 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:18.041436 9060 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:18.047631 9002 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:17.981537 8981 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.077106 5812 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.078260 4818 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:18.048014 8952 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.061757 8730 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:18.091157 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.073071 8962 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:18.100874 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 66788.3us
I20260806 04:08:18.048707 8997 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.109551 4992 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:18.117220 9108 log.cc:924] Time spent T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260806 04:08:18.072755 8757 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.121711 4810 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:18.120671 8780 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:08:18.125010 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 48124.1us
I20260806 04:08:18.108853 8950 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.121937 8952 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:18.141736 9001 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:18.142285 8952 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:18.142807 8952 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.143642 9001 consensus_queue.cc:1046] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:08:18.130777 9060 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.147980 9060 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:18.152056 9114 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260806 04:08:18.152585 5862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 693) took 4159 ms. Trace:
I20260806 04:08:18.156832 4996 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.120584 8955 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:18.121309 8750 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:18.175683 4996 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:18.153065 5862 rpcz_store.cc:276] 0806 04:08:13.993241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:13.993373 (+ 132us) service_pool.cc:227] Handling call
0806 04:08:17.398965 (+3405592us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:17.399284 (+ 319us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:08:17.399868 (+ 584us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:17.399896 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 04:08:17.400822 (+ 926us) log.cc:842] Serialized 78 byte log entry
0806 04:08:17.716737 (+315915us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:18.021106 (+304369us) log.cc:842] Serialized 14 byte log entry
0806 04:08:18.032481 (+ 11375us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.032581 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:18.136900 (+104319us) raft_consensus.cc:1671] finished
0806 04:08:18.152182 (+ 15282us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:18.152389 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26233,"prepare.queue_time_us":15206,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":17981,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":12487,"spinlock_wait_cycles":3413248,"thread_start_us":598987,"threads_started":2,"wal-append.queue_time_us":560343}
I20260806 04:08:18.195358 8955 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.196446 4996 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:18.197441 4996 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:18.197937 8955 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:18.200076 8532 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:18.200582 8532 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.204512 8955 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:18.175851 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.210372 4882 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:18.205024 8955 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:18.205821 8980 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.601s user 0.002s sys 0.010s
W20260806 04:08:18.220075 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 44317.7us
I20260806 04:08:18.207845 8780 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:18.232177 4882 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:18.238184 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:18.239140 4930 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.239598 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:18.219059 8955 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:18.228956 8936 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:18.252276 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.253391 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:18.247236 4879 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.265941 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.264081 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.267575 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.244668 8780 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:18.270725 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.279254 4882 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.245837 9001 consensus_queue.cc:1046] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260806 04:08:18.276654 8693 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:18.292737 8862 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:18.345870 8712 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:18.348268 8825 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:18.348789 8825 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:18.349155 8825 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.351208 8197 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:08:18.346168 8862 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:18.358875 4805 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:18.366284 8757 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:18.366919 8987 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:18.367412 8987 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:18.367810 8987 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:18.372928 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 447) took 1280 ms. Trace:
I20260806 04:08:18.373303 4817 rpcz_store.cc:276] 0806 04:08:17.092243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.538231 (+445988us) service_pool.cc:227] Handling call
0806 04:08:17.629069 (+ 90838us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:17.675009 (+ 45940us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.372710 (+697701us) spinlock_profiling.cc:243] Waited 90.4 ms on lock 0x7b5400271ad0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:18.372842 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207944064}
W20260806 04:08:18.374396 4813 consensus_meta.cc:287] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.000s
I20260806 04:08:18.377790 4813 raft_consensus.cc:1261] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:18.375583 4817 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 04:08:18.346082 9125 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.131s user 0.001s sys 0.001s
I20260806 04:08:18.280400 8780 raft_consensus.cc:686] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:18.352459 8641 raft_consensus.cc:3032] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:18.474987 8641 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:18.476322 8641 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.478250 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.283913 8936 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:18.289736 8180 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:08:18.423157 4817 raft_consensus.cc:1261] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20260806 04:08:18.431638 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1439) took 1701 ms (client timeout 1528 ms). Trace:
I20260806 04:08:18.484452 4817 pending_rounds.cc:76] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 2
W20260806 04:08:18.484535 4810 rpcz_store.cc:269] 0806 04:08:16.729733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:16.919319 (+189586us) service_pool.cc:227] Handling call
0806 04:08:18.431420 (+1512101us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002a3fd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:18.431557 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2762478336}
I20260806 04:08:18.444680 4930 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:18.432833 4804 raft_consensus.cc:1261] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:08:18.434638 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:18.486591 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:18.487285 8995 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:18.376654 8862 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:18.487864 8995 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.488022 4674 catalog_manager.cc:5495] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:18.286765 8730 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.489226 8936 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.466378 8873 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8.
I20260806 04:08:18.490553 8995 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:18.490808 9116 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260806 04:08:18.364353 8980 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:18.484975 4817 pending_rounds.cc:97] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 04:08:18.491412 4817 raft_consensus.cc:2866] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:18.491472 8980 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.490176 4674 catalog_manager.cc:5495] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:18.470715 9001 consensus_queue.cc:1046] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:08:18.473450 8532 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.471390 8825 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.510139 5854 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:18.511305 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1441) took 1570 ms. Trace:
I20260806 04:08:18.496632 8391 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:18.511888 4814 rpcz_store.cc:276] 0806 04:08:16.941378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.536550 (+595172us) service_pool.cc:227] Handling call
0806 04:08:17.549049 (+ 12499us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.567997 (+ 18948us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:17.568045 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:17.568086 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:17.568126 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.511023 (+942897us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54002706d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:18.511081 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:18.511231 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29214592}
I20260806 04:08:18.497704 4674 catalog_manager.cc:5495] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:18.401670 9054 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.535507 8520 raft_consensus.cc:3032] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:18.535809 8825 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.536154 8520 raft_consensus.cc:729] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:18.537678 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 2311) took 1275 ms. Trace:
I20260806 04:08:18.538120 4805 rpcz_store.cc:276] 0806 04:08:17.261916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.320112 (+ 58196us) service_pool.cc:227] Handling call
0806 04:08:18.537476 (+1217364us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54003c1c54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:18.537612 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2060613888}
I20260806 04:08:18.539515 4816 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:18.537405 8520 consensus_queue.cc:260] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.528146 4674 catalog_manager.cc:5495] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 3 to 4, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 4 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:18.526806 9001 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:18.524838 8391 raft_consensus.cc:686] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:18.558439 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 473724us
W20260806 04:08:18.552610 9047 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.847s user 0.011s sys 0.003s
I20260806 04:08:18.542634 8730 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:18.569032 4674 catalog_manager.cc:5495] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:18.573963 8456 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:18.576167 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.578186 8962 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:18.581063 8981 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.526252 9002 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20260806 04:08:18.557387 8520 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.592661 8987 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:18.595263 8921 consensus_meta.cc:287] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.633s user 0.012s sys 0.000s
I20260806 04:08:18.558370 5856 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:18.611892 9002 consensus_queue.cc:1225] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer d7c76b6dcd8945cd818b8107caf4a174 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:08:18.612082 8635 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:18.615236 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:18.623869 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:18.623991 8391 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.625475 5862 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:18.629781 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2327) took 1068 ms. Trace:
I20260806 04:08:18.630129 4804 rpcz_store.cc:276] 0806 04:08:17.560985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.987604 (+426619us) service_pool.cc:227] Handling call
0806 04:08:18.432445 (+444841us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:18.486244 (+ 53799us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.629598 (+143354us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b54002a3fd0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:18.629722 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":965697792}
I20260806 04:08:18.633172 5862 pending_rounds.cc:76] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Aborting all ops after (but not including) 5
I20260806 04:08:18.633575 5862 pending_rounds.cc:97] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260806 04:08:18.633829 5862 raft_consensus.cc:2866] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:18.633010 8997 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.556432 8975 consensus_queue.cc:1046] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
W20260806 04:08:18.639549 4992 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.015s sys 0.033s
W20260806 04:08:18.642369 4992 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.015s sys 0.033s
I20260806 04:08:18.645234 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:18.637470 8987 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.656956 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2328) took 1095 ms. Trace:
W20260806 04:08:18.653669 4995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.012s sys 0.024s
W20260806 04:08:18.659246 4995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.012s sys 0.024s
I20260806 04:08:18.604140 9001 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.658692 4816 rpcz_store.cc:276] 0806 04:08:17.564617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.990884 (+426267us) service_pool.cc:227] Handling call
0806 04:08:18.539186 (+548302us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:18.539877 (+ 691us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.654582 (+114705us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54003c1c50. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:18.656882 (+ 2300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1260126976}
I20260806 04:08:18.660847 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1685) took 5743 ms. Trace:
W20260806 04:08:18.660025 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 563722us
I20260806 04:08:18.647750 5853 raft_consensus.cc:1226] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:18.674347 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:18.661352 5030 rpcz_store.cc:276] 0806 04:08:12.917777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:13.021806 (+104029us) service_pool.cc:227] Handling call
0806 04:08:13.022119 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.725471 (+4703352us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.648090 (+922619us) spinlock_profiling.cc:243] Waited 4.61 s on lock 0x7b5400017a54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:18.660771 (+ 12681us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10612290816}
I20260806 04:08:18.673684 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:18.678795 4924 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.680006 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:18.680183 8997 raft_consensus.cc:3032] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:18.678223 4930 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
W20260806 04:08:18.681041 4808 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.024s sys 0.046s
I20260806 04:08:18.679404 4674 catalog_manager.cc:5495] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
W20260806 04:08:18.676286 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 586481us
W20260806 04:08:18.692317 4808 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.591s user 0.024s sys 0.046s
I20260806 04:08:18.638082 8635 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.642962 8981 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.028s
I20260806 04:08:18.703405 4992 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.703739 4674 catalog_manager.cc:5495] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 1 to 2, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 2 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:18.706919 9054 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.706188 4674 catalog_manager.cc:5495] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 reported cstate change: term changed from 3 to 5, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 5 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:18.672772 8532 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:18.708263 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.011s sys 0.023s
W20260806 04:08:18.713033 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.012s sys 0.023s
I20260806 04:08:18.712996 4924 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
W20260806 04:08:18.701665 8952 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.558s user 0.012s sys 0.000s
I20260806 04:08:18.716364 5812 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.641855 8780 consensus_queue.cc:237] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.691627 8997 raft_consensus.cc:729] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:18.717744 5863 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
W20260806 04:08:18.717468 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 520064us
I20260806 04:08:18.755959 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.734732 8730 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:18.719274 8980 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:18.760739 4809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.005s sys 0.011s
W20260806 04:08:18.764012 4809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.005s sys 0.011s
I20260806 04:08:18.743137 8532 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:18.743382 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.767393 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2312) took 1505 ms. Trace:
I20260806 04:08:18.770124 4809 rpcz_store.cc:276] 0806 04:08:17.262267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:17.615944 (+353677us) service_pool.cc:227] Handling call
0806 04:08:17.672903 (+ 56959us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:17.699922 (+ 27019us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:18.043168 (+343246us) log.cc:842] Serialized 14 byte log entry
0806 04:08:18.766910 (+723742us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:18.766982 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:18.767050 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:18.767165 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:18.767334 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":525630,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":4477,"thread_start_us":942165,"threads_started":2}
I20260806 04:08:18.763767 8825 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:18.743345 8952 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.797768 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:18.798640 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 531442us
I20260806 04:08:18.801111 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.800851 8825 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.801537 8921 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:18.802095 8921 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.802615 8952 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:18.803503 5867 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:18.804047 8921 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:18.780025 8532 raft_consensus.cc:686] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:18.809818 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:18.804407 8825 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:18.787276 9001 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:18.812968 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:18.814038 8197 consensus_queue.cc:1035] T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:08:18.814378 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.814869 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:08:18.814975 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:18.815447 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:18.817876 8641 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:18.788103 8997 consensus_queue.cc:260] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.820941 5859 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:18.826277 8863 consensus_queue.cc:1035] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:08:18.769361 8730 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.839049 4879 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:18.827711 4930 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.831442 8797 raft_consensus.cc:3032] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:18.842396 4804 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
W20260806 04:08:18.836458 8984 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.016s sys 0.032s
I20260806 04:08:18.848304 5816 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:18.846065 8797 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:18.860275 9139 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:18.860915 8797 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.858374 8921 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:18.863457 8921 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:18.863839 8921 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:18.865356 8635 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.827785 8180 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260806 04:08:18.850630 8862 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.868585 8862 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:18.869225 8862 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:08:18.872584 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:18.882431 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:18.853607 8984 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.651s user 0.016s sys 0.033s
I20260806 04:08:18.849139 9136 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:18.888846 8873 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260806 04:08:18.892522 8955 consensus_meta.cc:287] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.610s user 0.011s sys 0.001s
I20260806 04:08:18.848197 9028 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:18.897475 8797 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:18.898883 4996 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:18.895514 8921 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:18.902696 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 71430.4us
I20260806 04:08:18.845947 8975 consensus_queue.cc:1046] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260806 04:08:18.848474 8981 consensus_queue.cc:1046] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.055s
I20260806 04:08:18.904944 9028 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:18.952061 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 53255.8us
I20260806 04:08:18.946521 4930 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:18.962553 9028 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:18.900492 9002 consensus_queue.cc:1046] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 04:08:18.971166 5812 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.976037 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.977164 4813 raft_consensus.cc:2396] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 04:08:18.974645 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.980140 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.983479 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:18.984045 4809 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:08:18.985846 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:18.986796 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:18.987355 5037 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:18.989568 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:18.990193 5026 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:18.984079 4804 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:18.991717 4804 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:18.991832 5856 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:18.992532 4804 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:18.993109 8873 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:18.993526 4804 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:18.995362 8180 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:08:18.996306 8180 raft_consensus.cc:3032] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:18.996624 8180 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:18.997481 5029 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:18.997421 8180 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:18.999456 8180 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:18.993547 8873 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:19.000445 8873 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:18.986014 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:18.991312 8730 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:18.972673 9146 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 04:08:19.011089 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.019162 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:19.019843 5036 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:19.024149 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.025454 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.927037 8955 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.028484 8955 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:19.032446 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.033326 5029 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:18.865352 8532 consensus_queue.cc:237] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.034868 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.035712 5034 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 7: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:19.037416 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:18.904119 8520 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
W20260806 04:08:19.040498 4876 leader_election.cc:334] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.665s (SENT)
I20260806 04:08:19.041016 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:19.042117 9146 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:19.043274 4810 raft_consensus.cc:3032] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:19.043606 4810 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:19.044450 4810 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.045390 4810 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:19.046028 5036 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:19.046980 9002 raft_consensus.cc:3032] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:19.047405 9002 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:19.048218 9002 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.049441 9002 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:18.933820 8936 consensus_queue.cc:1035] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.119s
I20260806 04:08:19.058825 8391 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:19.059437 8391 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:19.059796 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.060640 8391 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.062783 5034 raft_consensus.cc:2396] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:19.063934 4882 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:18.977707 8975 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.069353 8975 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.071537 8975 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:18.974099 9094 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260806 04:08:18.978561 8981 raft_consensus.cc:3032] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:19.074267 8981 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:19.075311 8981 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.076438 8981 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:19.084931 4816 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:19.053565 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.100174 4804 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 04:08:19.094026 5028 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:19.112504 5037 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:19.084087 9146 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:08:19.133157 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.033s sys 0.060s
W20260806 04:08:19.212981 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.034s sys 0.060s
I20260806 04:08:19.132709 4809 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
W20260806 04:08:19.137687 5812 leader_election.cc:334] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.516s (SENT)
I20260806 04:08:19.147378 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 454) took 1438 ms. Trace:
I20260806 04:08:19.214391 4808 rpcz_store.cc:276] 0806 04:08:17.708731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:18.037812 (+329081us) service_pool.cc:227] Handling call
0806 04:08:18.056266 (+ 18454us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:18.074225 (+ 17959us) raft_consensus.cc:1520] Early marking committed up to index 10
0806 04:08:18.692853 (+618628us) log.cc:842] Serialized 14 byte log entry
0806 04:08:19.146953 (+454100us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:19.147024 (+ 71us) raft_consensus.cc:1631] Marking committed up to 10
0806 04:08:19.147081 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:19.147158 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:19.147308 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16257,"prepare.queue_time_us":176281,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":19979,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5138,"thread_start_us":1028891,"threads_started":2}
W20260806 04:08:19.149217 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 448125us
I20260806 04:08:19.151573 5036 raft_consensus.cc:1261] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:19.155694 8962 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:19.162216 8997 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:19.172551 8532 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.139973 8980 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:19.150550 8921 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:19.151201 8995 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.226991 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.184098 4810 raft_consensus.cc:1261] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260806 04:08:19.155889 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 452549us
I20260806 04:08:19.184847 8936 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.228745 5034 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:08:19.225785 8532 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.232820 8995 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.223402 8780 consensus_queue.cc:1046] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260806 04:08:19.234643 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
W20260806 04:08:19.242887 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 115371us
I20260806 04:08:19.241389 8921 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.267532 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:19.267740 8921 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.272917 4992 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:19.267854 8980 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.287616 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.236716 8532 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.235204 8812 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:19.239385 8995 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.272428 5857 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:19.301051 4882 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:19.304052 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.304823 4923 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:19.311030 4882 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:19.345264 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:19.291190 9094 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:19.349340 4882 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:19.354862 4882 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.360558 9094 raft_consensus.cc:686] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:19.361611 9094 consensus_queue.cc:237] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.364717 9153 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:19.365355 9153 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.367689 9153 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:19.369997 5029 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 04:08:19.287030 8862 consensus_queue.cc:1035] T 98064c6d64f74659bfa102bd5a341635 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260806 04:08:19.371079 4809 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:08:19.236364 8936 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.384876 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.385569 5038 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:19.388558 4992 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:19.353492 9146 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:19.398088 9150 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260806 04:08:19.399448 4923 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:19.400269 4923 raft_consensus.cc:595] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:19.403115 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:19.403411 9159 log.cc:924] Time spent T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260806 04:08:19.403810 5035 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
W20260806 04:08:19.404877 9155 log.cc:924] Time spent T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260806 04:08:19.405236 4920 raft_consensus.cc:1226] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:19.355857 8750 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:19.413710 4927 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:19.409825 8980 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.306609 8812 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.411002 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.411805 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.428308 4926 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:19.412468 8936 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.412874 4928 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:19.373102 8995 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:19.381052 8641 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:19.435603 4923 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:19.376935 8635 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.442202 4804 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260806 04:08:19.443641 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.393049 8975 raft_consensus.cc:3032] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:19.455190 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.456332 4804 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
I20260806 04:08:19.358109 8873 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.458806 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.507378 5029 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:19.508893 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.461198 4920 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:19.510463 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.512909 5029 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
W20260806 04:08:19.514523 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 71064.9us
I20260806 04:08:19.393898 8825 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260806 04:08:19.433545 9153 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:19.445083 8641 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:19.523317 4927 raft_consensus.cc:1226] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:19.441327 8520 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:19.523042 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.534857 5029 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 7: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:19.414599 9130 consensus_queue.cc:1035] T da2e93c822734c02a26703dabb801932 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.097s
W20260806 04:08:19.529829 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 153576us
I20260806 04:08:19.447227 8975 raft_consensus.cc:729] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:19.564812 4882 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.568858 4920 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:19.572862 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.573789 5029 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 2.
W20260806 04:08:19.522572 4746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 64227.3us
I20260806 04:08:19.523720 8780 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260806 04:08:19.575026 8906 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.580083 4882 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.586966 4920 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 04:08:19.592636 8533 raft_consensus.cc:2781] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:19.594410 8812 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.582682 8936 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260806 04:08:19.595443 8975 consensus_queue.cc:260] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.596033 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.596645 8975 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:19.596698 4808 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:08:19.582859 9146 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.015s
I20260806 04:08:19.597648 9001 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:08:19.598042 9001 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:19.598397 9001 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:19.527979 4919 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:19.579177 8906 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.561977 5030 raft_consensus.cc:1226] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:19.607254 8906 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.536506 8641 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:19.612170 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:19.614089 4768 connection.cc:232] RPC call timeout handler was delayed by -4.7377821539994329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
I20260806 04:08:19.506502 9028 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.619166 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:19.621454 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 268690us
I20260806 04:08:19.458639 8635 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.564356 8955 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260806 04:08:19.637768 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.650507 4804 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:19.536392 9153 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.651760 4924 raft_consensus.cc:2379] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:19.578339 8520 consensus_queue.cc:1035] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260806 04:08:19.655105 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.655962 4920 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:19.639643 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:19.657841 5029 raft_consensus.cc:2362] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5. Re-sending same reply.
I20260806 04:08:19.649297 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.658705 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.659564 4818 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:19.616148 8921 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.667922 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:19.668694 5029 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:19.670923 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:19.673892 4808 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 3.
I20260806 04:08:19.671685 5026 raft_consensus.cc:2362] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8. Re-sending same reply.
I20260806 04:08:19.678105 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:19.678865 5029 raft_consensus.cc:3037] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:19.678977 5816 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.679481 5862 raft_consensus.cc:1226] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
I20260806 04:08:19.704141 4882 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.688755 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.670611 5857 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
W20260806 04:08:19.731705 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 204509us
W20260806 04:08:19.730201 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 93109us
I20260806 04:08:19.710804 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:19.714094 8863 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:19.713065 8962 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:19.737797 5029 raft_consensus.cc:2454] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:19.724725 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.746994 5029 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260806 04:08:19.733026 8797 consensus_meta.cc:287] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.806s user 0.008s sys 0.006s
I20260806 04:08:19.749495 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.750595 5029 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:19.754024 4920 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:19.757426 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:19.761756 5029 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:19.738301 8391 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:19.771700 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 121651us
I20260806 04:08:19.741457 8962 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:19.741258 8863 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:19.737351 8533 raft_consensus.cc:686] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:19.709612 9191 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260806 04:08:19.739442 9146 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:19.784997 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 106088us
I20260806 04:08:19.786461 4876 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:19.788383 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46116.3us
I20260806 04:08:19.788429 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.789278 4919 raft_consensus.cc:2454] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:19.791484 9153 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:19.792294 9184 log.cc:924] Time spent T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
W20260806 04:08:19.795029 9170 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
W20260806 04:08:19.795135 9168 log.cc:924] Time spent T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260806 04:08:19.777907 5853 raft_consensus.cc:2354] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5.
W20260806 04:08:19.796408 9199 log.cc:924] Time spent T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 04:08:19.800060 9153 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.800660 9153 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.772647 8391 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:19.804309 8635 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.804570 8391 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:19.804260 9153 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:19.806107 8981 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.729s user 0.002s sys 0.009s
I20260806 04:08:19.797658 5816 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.808820 8635 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:19.809222 8635 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:19.809603 8635 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:19.811276 9178 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260806 04:08:19.811991 9156 raft_consensus.cc:3032] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:19.812414 9156 raft_consensus.cc:729] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:19.813287 9156 consensus_queue.cc:260] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:19.816177 9149 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.588s user 0.007s sys 0.000s
W20260806 04:08:19.818881 4768 leader_election.cc:334] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.050s (SENT)
W20260806 04:08:19.819886 4768 connection.cc:232] RPC call timeout handler was delayed by -4.6396821539383382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
W20260806 04:08:19.820592 4768 connection.cc:232] RPC call timeout handler was delayed by -4.5181821540463716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
I20260806 04:08:19.821909 8975 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.778729 8863 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:19.822474 8975 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.825093 8975 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:19.825922 4768 connection.cc:232] RPC call timeout handler was delayed by -3.2474314861465245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
I20260806 04:08:19.828717 4770 leader_election.cc:398] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5.
I20260806 04:08:19.829094 4770 leader_election.cc:401] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:19.828979 8180 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:19.830302 9153 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:19.831852 9153 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.828153 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:19.835718 9177 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 04:08:19.836714 9209 consensus_queue.cc:1035] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:08:19.788790 8921 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.838001 8921 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.839972 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.769172 8873 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:19.840094 8921 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:19.844355 4995 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:19.890938 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:19.847223 5859 raft_consensus.cc:1226] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:19.862910 5816 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:19.848984 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:19.851547 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.897419 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:19.843086 4992 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:19.892139 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:19.828758 4768 connection.cc:232] RPC call timeout handler was delayed by -2.8667314860504121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
W20260806 04:08:19.907756 4768 connection.cc:232] RPC call timeout handler was delayed by -2.8244314859621227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2247s for a response
W20260806 04:08:19.908092 4768 connection.cc:232] RPC call timeout handler was delayed by -2.8226314859930426s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
I20260806 04:08:19.908111 4926 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
W20260806 04:08:19.908463 4768 connection.cc:232] RPC call timeout handler was delayed by -2.8163314859848469s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2256s for a response
W20260806 04:08:19.908751 4768 connection.cc:232] RPC call timeout handler was delayed by -1.8677314859814942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.331s for a response
I20260806 04:08:19.908818 4926 raft_consensus.cc:595] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:19.909035 4768 connection.cc:232] RPC call timeout handler was delayed by -1.8677314859814942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.331s for a response
I20260806 04:08:19.909261 4926 raft_consensus.cc:603] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:19.841243 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:19.843978 4876 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:19.890517 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:19.898344 4931 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:19.968374 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:19.972513 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 128229us
I20260806 04:08:19.835708 9028 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:19.983762 9028 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:08:19.787246 8825 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:19.849459 5857 raft_consensus.cc:1261] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260806 04:08:19.969395 4817 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:20.001111 9028 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:20.002971 9028 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:19.986797 4813 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:08:19.979883 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:19.842253 9153 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:19.988620 4923 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260806 04:08:19.992605 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:19.831912 8520 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:19.993283 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:19.843117 9002 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.793s user 0.014s sys 0.000s
I20260806 04:08:19.846406 8975 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:08:19.814636 9156 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:19.910764 9195 raft_consensus.cc:2741] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:08:19.843173 8533 consensus_queue.cc:237] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:20.000774 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 157014us
I20260806 04:08:19.988099 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:20.021032 5036 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:20.012006 5028 raft_consensus.cc:2362] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6. Re-sending same reply.
I20260806 04:08:19.985796 9139 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 04:08:19.846201 8997 consensus_meta.cc:287] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.615s user 0.009s sys 0.000s
W20260806 04:08:20.024963 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 181925us
I20260806 04:08:20.026620 4920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
W20260806 04:08:20.029423 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 231629us
I20260806 04:08:20.029443 4920 raft_consensus.cc:595] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:20.032899 4920 raft_consensus.cc:603] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:19.840934 8635 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.038916 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:20.042039 5812 leader_election.cc:334] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.831s (SENT)
I20260806 04:08:20.015054 8862 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:20.015419 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:19.994796 9001 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:19.850806 9146 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:19.991598 8962 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:20.061093 5812 leader_election.cc:334] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.909s (SENT)
I20260806 04:08:20.061563 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:20.073166 5812 leader_election.cc:334] T 141037797aa84869bb31e99383874122 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.214s (SENT)
W20260806 04:08:20.073290 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 161341us
I20260806 04:08:20.018206 9130 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260806 04:08:20.038231 9139 raft_consensus.cc:686] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:20.049265 8981 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:08:20.081925 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 57805.6us
I20260806 04:08:20.107831 4920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:20.078545 5812 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:20.118377 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 83304.1us
I20260806 04:08:20.119858 4927 raft_consensus.cc:2379] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:20.078989 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:20.124902 4920 raft_consensus.cc:595] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.125284 4920 raft_consensus.cc:603] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:20.125334 4928 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:20.023494 8520 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:20.058928 8797 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:20.127887 5862 raft_consensus.cc:1226] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:20.065222 8862 raft_consensus.cc:686] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
W20260806 04:08:20.129817 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 69153.3us
W20260806 04:08:20.150835 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 72442.5us
W20260806 04:08:20.124528 9199 log.cc:924] Time spent T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260806 04:08:20.077646 8641 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:20.152252 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:20.157367 4992 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.127346 8750 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:20.161648 4810 raft_consensus.cc:1261] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260806 04:08:20.162381 4768 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:20.165808 5812 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.152952 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:20.155215 8532 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:20.176729 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:20.164004 8906 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:20.165452 9204 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:20.175061 8997 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:20.175246 5856 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:20.175283 8952 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:08:20.150692 8797 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.175786 9215 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.023s
W20260806 04:08:20.195055 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 43824.3us
I20260806 04:08:20.243968 4992 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.183928 8812 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:20.182541 8906 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:20.179699 8532 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:20.190097 8984 consensus_queue.cc:1046] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20260806 04:08:20.167742 8862 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:20.252117 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 231434us
I20260806 04:08:20.253818 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:20.236591 9216 log.cc:924] Time spent T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.219s user 0.006s sys 0.000s
I20260806 04:08:20.250970 9130 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:20.242051 8952 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:20.246377 8812 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:20.251863 8906 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:20.253232 8635 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:20.287513 8641 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:20.296474 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.185457 9139 consensus_queue.cc:237] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.259357 9001 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:20.267599 8962 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:20.225426 9232 consensus_queue.cc:1035] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
I20260806 04:08:20.283601 8520 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:20.327440 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 83409.8us
I20260806 04:08:20.267228 9191 consensus_queue.cc:1035] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
W20260806 04:08:20.322901 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 146109us
I20260806 04:08:20.292913 8812 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:20.335651 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:20.291973 9146 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:20.321273 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 308383us
I20260806 04:08:20.364006 9146 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:20.354318 9001 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.054s
I20260806 04:08:20.258126 8532 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:20.354202 8863 consensus_meta.cc:287] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.531s user 0.012s sys 0.002s
W20260806 04:08:20.412778 9220 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20260806 04:08:20.412184 8863 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:20.416141 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 64247.2us
W20260806 04:08:20.418980 9226 log.cc:924] Time spent T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.217s user 0.002s sys 0.000s
I20260806 04:08:20.294979 8952 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.405968 9146 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:20.427153 9146 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:20.428341 9236 log.cc:924] Time spent T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
I20260806 04:08:20.421481 9001 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:20.436528 9001 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:20.365638 4876 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:20.416767 9224 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.214s user 0.002s sys 0.000s
W20260806 04:08:20.453753 9241 log.cc:924] Time spent T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260806 04:08:20.460616 4992 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.466946 9001 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:20.479434 9204 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:20.284454 8391 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.513803 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:20.518253 4818 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:08:20.542738 9002 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:20.531832 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:20.548834 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 314679us
I20260806 04:08:20.549880 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:20.484766 9242 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:20.545655 9002 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.562290 9153 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:20.438131 9231 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 0.207s user 0.004s sys 0.000s
I20260806 04:08:20.554626 8980 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:08:20.601589 8980 raft_consensus.cc:686] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:08:20.602451 9028 consensus_meta.cc:287] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.599s user 0.015s sys 0.011s
I20260806 04:08:20.556077 9169 raft_consensus.cc:3032] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:20.607095 9169 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:08:20.583832 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 218300us
W20260806 04:08:20.550392 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 296668us
W20260806 04:08:20.610693 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 192050us
I20260806 04:08:20.598738 9153 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.553224 8984 consensus_queue.cc:1046] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.068s
I20260806 04:08:20.596837 4813 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:20.614526 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:20.649333 8797 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:20.655869 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 46225.1us
I20260806 04:08:20.640242 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:20.625999 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:20.701983 4805 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:20.657819 4876 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:20.706259 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 48644.6us
I20260806 04:08:20.614046 4992 leader_election.cc:398] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:20.711220 4992 leader_election.cc:401] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:08:20.715281 5859 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
W20260806 04:08:20.727702 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 113526us
I20260806 04:08:20.700264 4814 pending_rounds.cc:76] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 6
I20260806 04:08:20.731695 4814 pending_rounds.cc:97] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 04:08:20.733582 4814 raft_consensus.cc:2866] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:20.740072 4876 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.744841 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:20.756786 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:20.812103 4927 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:20.729719 8995 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.012s sys 0.016s
I20260806 04:08:20.537613 8641 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:20.631129 8997 raft_consensus.cc:3032] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:20.807622 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:20.645356 9221 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.419s user 0.007s sys 0.000s
I20260806 04:08:20.604265 9028 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:20.827301 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 97120.7us
I20260806 04:08:20.865006 4930 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:20.639005 9002 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:20.619692 8863 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:20.617116 8952 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:20.835840 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.009s sys 0.034s
W20260806 04:08:20.933701 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.781s user 0.009s sys 0.035s
W20260806 04:08:20.935680 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 783820us
W20260806 04:08:20.834144 8995 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.667s user 0.014s sys 0.016s
W20260806 04:08:20.836736 4924 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.000s sys 0.043s
I20260806 04:08:20.794814 8984 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:08:20.939389 4924 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.000s sys 0.043s
W20260806 04:08:20.849283 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 109550us
I20260806 04:08:20.885464 8180 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 04:08:20.922323 9234 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.013s sys 0.061s
I20260806 04:08:20.966775 4818 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:20.950076 4876 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:20.642269 9242 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:20.863695 8997 raft_consensus.cc:729] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:21.039323 4919 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:08:20.958204 9234 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.705s user 0.015s sys 0.061s
I20260806 04:08:20.833657 8980 consensus_queue.cc:237] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:21.073933 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 124384us
I20260806 04:08:20.976346 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:21.048543 8180 raft_consensus.cc:686] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:21.095924 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2359) took 1393 ms. Trace:
I20260806 04:08:21.093662 5853 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:21.098623 4804 rpcz_store.cc:276] 0806 04:08:19.702515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:19.702673 (+ 158us) service_pool.cc:227] Handling call
0806 04:08:19.703083 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:19.719825 (+ 16742us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:19.719871 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:19.730486 (+ 10615us) log.cc:842] Serialized 26 byte log entry
0806 04:08:19.850939 (+120453us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b4800261558. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23de8017f1 00007f23de7f88e7 00007f23de7f877a 00007f23de8ec95f 00007f23de8ec627 00007f23de900b2c 00007f23de900ad8 00007f23de900a58 00007f23de900a13 00007f23de8ffcbc 00007f23dd881be6
0806 04:08:19.995115 (+144176us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:19.995172 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:19.995262 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:20.644186 (+648924us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:21.095634 (+451448us) raft_consensus.cc:1671] finished
0806 04:08:21.095656 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.095821 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10356,"raft.queue_time_us":33985,"raft.run_cpu_time_us":2581,"raft.run_wall_time_us":110719,"spinlock_wait_cycles":10367616,"thread_start_us":248265,"threads_started":1,"wal-append.queue_time_us":299076}
I20260806 04:08:21.102955 8955 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.111651 9028 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:21.111528 8955 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.113991 9242 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:21.101686 9206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.009s sys 0.026s
W20260806 04:08:21.116245 4817 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.953s user 0.068s sys 0.135s
W20260806 04:08:21.116768 4817 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s user 0.068s sys 0.136s
I20260806 04:08:20.798699 9169 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.068150 8984 consensus_queue.cc:1046] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 04:08:21.114336 8955 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:21.155038 5856 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.923s user 0.020s sys 0.011s
I20260806 04:08:21.184582 5856 raft_consensus.cc:2454] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:21.185416 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 364) took 1111 ms. Trace:
I20260806 04:08:21.185824 5856 rpcz_store.cc:276] 0806 04:08:20.074522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.107729 (+ 33207us) service_pool.cc:227] Handling call
0806 04:08:21.185333 (+1077604us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:21.175797 9208 log.cc:924] Time spent T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 1.380s user 0.011s sys 0.003s
I20260806 04:08:21.188818 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2350) took 1741 ms. Trace:
I20260806 04:08:21.087211 9153 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:21.115097 9206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.040s user 0.009s sys 0.027s
I20260806 04:08:21.193790 8391 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:21.104017 8780 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.151638 8750 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 04:08:21.195257 8780 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:21.115501 9139 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.012s sys 0.019s
I20260806 04:08:21.101640 9186 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.196851 9186 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.197777 8780 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:21.186738 9245 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
W20260806 04:08:21.199342 9139 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.013s sys 0.020s
I20260806 04:08:21.202070 9186 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:21.202241 8812 consensus_meta.cc:287] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.723s user 0.014s sys 0.010s
W20260806 04:08:21.203433 9256 log.cc:924] Time spent T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260806 04:08:21.181007 5814 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.006s sys 0.069s
I20260806 04:08:21.181407 8180 consensus_queue.cc:237] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:21.205861 5814 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.798s user 0.006s sys 0.069s
I20260806 04:08:21.188232 8997 consensus_queue.cc:260] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:21.172785 9130 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.013s sys 0.022s
I20260806 04:08:21.189549 9169 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:21.099085 9259 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:21.189211 4816 rpcz_store.cc:276] 0806 04:08:19.447747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:19.447990 (+ 243us) service_pool.cc:227] Handling call
0806 04:08:19.448335 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:19.448793 (+ 458us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:19.448829 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:19.506206 (+ 57377us) log.cc:842] Serialized 26 byte log entry
0806 04:08:19.705655 (+199449us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:19.705692 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:19.705760 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:21.096585 (+1390825us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b24001bbf48. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d688acf4 00007f23d6888dfb 00007f23d6917559 00007f23d69171e5 00007f23d691682d 00007f23d95c546e 00007f23d95dcc0c 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551
0806 04:08:21.188475 (+ 91890us) raft_consensus.cc:1671] finished
0806 04:08:21.188501 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.188754 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":596,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":55573,"spinlock_wait_cycles":151862144,"thread_start_us":136836,"threads_started":1,"wal-append.queue_time_us":276275}
I20260806 04:08:21.203176 8812 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:21.202085 9146 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.774s user 0.015s sys 0.005s
W20260806 04:08:21.218951 9130 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.569s user 0.013s sys 0.022s
I20260806 04:08:21.234249 9259 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:21.244019 9001 consensus_meta.cc:287] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.774s user 0.025s sys 0.000s
W20260806 04:08:21.263918 4808 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.004s sys 0.013s
I20260806 04:08:21.267508 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.226310 9146 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.254465 9139 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:21.287667 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 673939us
I20260806 04:08:21.290443 4992 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:21.290934 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 222267us
W20260806 04:08:21.294836 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 457788us
W20260806 04:08:21.293604 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 217857us
W20260806 04:08:21.283519 4996 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.729s (SENT) [suppressed 47 similar messages]
I20260806 04:08:21.317174 4992 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:21.337081 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 43687.7us
I20260806 04:08:21.288436 8520 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:21.365660 4992 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:21.370894 9259 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:21.373778 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.002s sys 0.018s
I20260806 04:08:21.264236 9001 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:21.341747 4993 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.500s user 0.013s sys 0.009s
I20260806 04:08:21.288089 9139 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.348170 5862 raft_consensus.cc:1226] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:21.351130 9245 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:21.351827 8780 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:21.361588 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 488) took 1301 ms. Trace:
I20260806 04:08:21.404454 4919 rpcz_store.cc:276] 0806 04:08:20.060425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.061222 (+ 797us) service_pool.cc:227] Handling call
0806 04:08:20.078564 (+ 17342us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:21.044752 (+966188us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.361273 (+316521us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54001b8ad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:21.361464 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2189682560}
W20260806 04:08:21.363739 5030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2144) took 1460 ms (client timeout 1680 ms). Trace:
I20260806 04:08:21.364562 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:21.294539 8975 consensus_meta.cc:287] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 1.270s user 0.020s sys 0.003s
W20260806 04:08:21.374249 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.574s user 0.003s sys 0.019s
W20260806 04:08:21.386886 4927 consensus_meta.cc:287] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.558s user 0.013s sys 0.002s
W20260806 04:08:21.406128 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 63560.7us
W20260806 04:08:21.406083 5030 rpcz_store.cc:269] 0806 04:08:19.903224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:19.903362 (+ 138us) service_pool.cc:227] Handling call
0806 04:08:21.363411 (+1460049us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003205d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:21.363613 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2859768064}
I20260806 04:08:21.366079 8520 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:21.411597 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 611164us
I20260806 04:08:21.411615 4927 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:21.379469 9204 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:08:21.418073 5816 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.421627 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.428275 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:21.429128 4818 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:21.411499 9231 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.007s sys 0.031s
I20260806 04:08:21.390611 8862 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:21.422987 9269 log.cc:924] Time spent T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260806 04:08:21.435921 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:21.446686 5812 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:21.450618 8635 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 04:08:21.456020 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 58569.5us
I20260806 04:08:21.448798 8921 raft_consensus.cc:2781] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:21.460660 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:21.446197 8906 consensus_meta.cc:287] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.122s user 0.018s sys 0.004s
W20260806 04:08:21.440026 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 540551us
I20260806 04:08:21.420738 8797 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:21.421785 8520 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:21.471056 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.421023 8975 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5.
I20260806 04:08:21.482112 4879 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:21.434890 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 67754us
I20260806 04:08:21.474305 4768 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:21.484298 4992 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:21.426609 5856 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:21.432094 9283 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.490033 4818 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:21.498953 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:21.423771 9028 consensus_queue.cc:1046] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20260806 04:08:21.506987 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:21.511307 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.512364 4805 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
W20260806 04:08:21.453939 9231 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.008s sys 0.032s
I20260806 04:08:21.491721 4992 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:21.499719 4818 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:21.503510 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.516170 4804 raft_consensus.cc:2396] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260806 04:08:21.506016 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:21.519374 9209 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.520704 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:21.520383 9209 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.521770 5853 raft_consensus.cc:2396] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260806 04:08:21.509564 4770 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.470400 8635 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:21.519533 9173 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:21.485453 9245 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.430869 8780 consensus_queue.cc:1046] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260806 04:08:21.543862 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.558984 4805 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:21.502278 9195 raft_consensus.cc:2781] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:21.497995 8921 raft_consensus.cc:686] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:21.488296 9278 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.566926 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 668) took 1537 ms. Trace:
I20260806 04:08:21.568634 5859 rpcz_store.cc:276] 0806 04:08:20.030061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.030644 (+ 583us) service_pool.cc:227] Handling call
0806 04:08:20.050916 (+ 20272us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:20.727318 (+676402us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.566476 (+839158us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b54004448d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:21.566808 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1526270208}
I20260806 04:08:21.471414 8862 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:21.580302 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 88692us
I20260806 04:08:21.581292 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:21.581514 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 99553.9us
I20260806 04:08:21.454875 9276 consensus_queue.cc:1046] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260806 04:08:21.534830 8635 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:21.537703 8981 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:21.557790 8812 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:21.595441 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:21.609056 4919 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:21.596817 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:21.579214 4809 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:21.610752 4930 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:21.532923 9285 raft_consensus.cc:2726] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:21.594944 9130 consensus_queue.cc:1046] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260806 04:08:21.482957 9251 consensus_queue.cc:1046] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260806 04:08:21.489022 8906 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:21.626811 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 143540us
I20260806 04:08:21.632885 4992 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:21.589639 9195 raft_consensus.cc:686] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:21.651466 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 367942us
I20260806 04:08:21.592429 9273 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.585999 9139 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:21.603881 9207 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:21.586757 8936 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:21.587142 9146 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:21.659988 5816 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.670817 4818 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:21.596010 8981 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:21.653955 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:21.636457 9267 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.354s user 0.001s sys 0.001s
I20260806 04:08:21.625649 8863 raft_consensus.cc:2781] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:21.712436 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.570048 9245 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.722394 4818 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:21.724611 9275 raft_consensus.cc:3032] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:21.725214 9275 raft_consensus.cc:729] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
W20260806 04:08:21.725489 9281 log.cc:924] Time spent T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260806 04:08:21.728487 9245 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:21.731079 8862 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:21.700613 9291 raft_consensus.cc:3032] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:21.731980 9291 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:21.734540 9284 log.cc:924] Time spent T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.239s user 0.001s sys 0.000s
I20260806 04:08:21.690302 9173 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.735965 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 481) took 1616 ms. Trace:
I20260806 04:08:21.736477 4814 rpcz_store.cc:276] 0806 04:08:20.119863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.572458 (+452595us) service_pool.cc:227] Handling call
0806 04:08:20.699079 (+126621us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:20.735012 (+ 35933us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:20.778276 (+ 43264us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:20.830505 (+ 52229us) log.cc:842] Serialized 26 byte log entry
0806 04:08:21.366414 (+535909us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:21.366475 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.366589 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:21.735672 (+369083us) raft_consensus.cc:1671] finished
0806 04:08:21.735702 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.735879 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":350533,"threads_started":1,"wal-append.queue_time_us":559057}
I20260806 04:08:21.739773 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 487) took 1679 ms. Trace:
I20260806 04:08:21.740526 8997 raft_consensus.cc:3037] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:21.742117 4814 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:21.742944 4809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1462) took 1981 ms (client timeout 1729 ms). Trace:
W20260806 04:08:21.745617 4809 rpcz_store.cc:269] 0806 04:08:19.761039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:19.784609 (+ 23570us) service_pool.cc:227] Handling call
0806 04:08:21.736037 (+1951428us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540040f954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:21.742863 (+ 6826us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3676825856}
I20260806 04:08:21.744271 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:21.752274 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 99926.6us
I20260806 04:08:21.689759 9204 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:21.692025 9001 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:21.754176 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:21.751425 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 91559.9us
I20260806 04:08:21.756568 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.763571 4809 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:21.740211 4924 rpcz_store.cc:276] 0806 04:08:20.060081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.061394 (+ 1313us) service_pool.cc:227] Handling call
0806 04:08:20.078330 (+ 16936us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:20.942254 (+863924us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:20.942445 (+ 191us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:21.031204 (+ 88759us) log.cc:842] Serialized 26 byte log entry
0806 04:08:21.417248 (+386044us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:21.417299 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.417400 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:21.739540 (+322140us) raft_consensus.cc:1671] finished
0806 04:08:21.739565 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.739709 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72673,"raft.queue_time_us":208451,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":138,"thread_start_us":1119617,"threads_started":2,"wal-append.queue_time_us":610180}
W20260806 04:08:21.759883 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 127170us
W20260806 04:08:21.744441 9169 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.007s
W20260806 04:08:21.720364 9156 consensus_meta.cc:287] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.697s user 0.017s sys 0.011s
I20260806 04:08:21.760927 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.581157 9278 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.713411 8981 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:21.770251 4996 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.732925 9291 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.778779 9291 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:21.726267 9275 consensus_queue.cc:260] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.780514 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 367) took 1035 ms. Trace:
I20260806 04:08:21.736474 9173 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:21.515789 9283 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.759900 9153 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:21.705523 9273 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.792362 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:21.760179 8532 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:21.765592 8780 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:21.760988 9209 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:21.790447 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 316247us
I20260806 04:08:21.758033 9207 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.779932 9275 raft_consensus.cc:3037] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:21.834872 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 80790.8us
I20260806 04:08:21.862706 4876 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:21.715342 8936 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.865638 8936 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:21.871371 4809 raft_consensus.cc:2396] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 04:08:21.834154 4930 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:21.789232 5857 rpcz_store.cc:276] 0806 04:08:20.745368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.745882 (+ 514us) service_pool.cc:227] Handling call
0806 04:08:20.814110 (+ 68228us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:21.177057 (+362947us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:21.177120 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:21.178184 (+ 1064us) log.cc:842] Serialized 26 byte log entry
0806 04:08:21.449248 (+271064us) raft_consensus.cc:1631] Marking committed up to 10
0806 04:08:21.449291 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.449362 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:21.780250 (+330888us) raft_consensus.cc:1671] finished
0806 04:08:21.780271 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.780417 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26449,"raft.queue_time_us":124276,"raft.run_cpu_time_us":745,"raft.run_wall_time_us":54979,"thread_start_us":612764,"threads_started":2,"wal-append.queue_time_us":575688}
I20260806 04:08:21.780386 9245 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:21.788964 9139 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:21.836892 9242 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:21.848351 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1469) took 1351 ms. Trace:
I20260806 04:08:21.879112 4808 rpcz_store.cc:276] 0806 04:08:20.497913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.650910 (+152997us) service_pool.cc:227] Handling call
0806 04:08:20.651443 (+ 533us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:21.280724 (+629281us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:21.280780 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:21.286469 (+ 5689us) log.cc:842] Serialized 26 byte log entry
0806 04:08:21.690199 (+403730us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:21.690249 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.690339 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:21.848093 (+157754us) raft_consensus.cc:1671] finished
0806 04:08:21.848117 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.848282 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75854,"raft.queue_time_us":313641,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":47447,"thread_start_us":854814,"threads_started":2,"wal-append.queue_time_us":509967}
I20260806 04:08:21.839013 8532 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:21.852555 8906 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:21.796609 8750 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:21.855343 9296 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
W20260806 04:08:21.856431 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 997e4d9ff3ca465199a888e8af23ed7b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:21.789311 9157 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.838999 8863 raft_consensus.cc:686] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:21.742504 8921 consensus_queue.cc:237] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.844003 9234 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.836503 9153 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:08:21.907730 4651 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 51440.7us
W20260806 04:08:21.914906 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 52356.1us
I20260806 04:08:21.874500 9139 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.919499 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.876286 8780 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:21.902705 8180 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.039s sys 0.026s
I20260806 04:08:21.874262 9245 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.879027 9242 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:21.906860 8532 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:21.945585 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:21.947446 5026 raft_consensus.cc:3037] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:21.949417 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:21.949647 5030 raft_consensus.cc:1261] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:08:21.949858 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1463) took 2188 ms. Trace:
W20260806 04:08:21.939260 8180 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s user 0.039s sys 0.026s
I20260806 04:08:21.917943 4805 raft_consensus.cc:1261] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:21.935750 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.235818 5029 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:21.907640 8984 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:21.939977 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:21.944981 4882 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:21.934478 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.237618 5038 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:08:21.954238 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 349356us
I20260806 04:08:21.950218 4817 rpcz_store.cc:276] 0806 04:08:19.761351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:20.002047 (+240696us) service_pool.cc:227] Handling call
0806 04:08:20.002514 (+ 467us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:20.003016 (+ 502us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:08:20.105202 (+102186us) log.cc:842] Serialized 14 byte log entry
0806 04:08:21.117339 (+1012137us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:21.172485 (+ 55146us) log.cc:842] Serialized 26 byte log entry
0806 04:08:21.172697 (+ 212us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:21.405526 (+232829us) log.cc:842] Serialized 14 byte log entry
0806 04:08:21.419763 (+ 14237us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:21.420001 (+ 238us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:21.949628 (+529627us) raft_consensus.cc:1671] finished
0806 04:08:21.949653 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:21.949796 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27885,"prepare.queue_time_us":678857,"prepare.run_cpu_time_us":425,"prepare.run_wall_time_us":10190,"raft.queue_time_us":207641,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":4660,"thread_start_us":1244485,"threads_started":3,"wal-append.queue_time_us":1095875}
I20260806 04:08:21.956386 5035 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:21.902637 9169 consensus_queue.cc:1046] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 04:08:21.836087 9195 consensus_queue.cc:237] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.917217 9157 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.958915 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:21.916934 8750 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.228025 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.278772 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.228776 4809 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:08:22.025075 9243 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.954838 8641 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:22.122388 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.286005 5030 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:21.941541 9002 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:22.055760 5026 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:21.939725 9139 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:21.921425 9293 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:21.869275 9207 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.303884 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.231418 9322 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:21.945721 9242 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:08:22.266708 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 326799us
I20260806 04:08:21.942342 8780 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:21.957281 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.325106 5034 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:22.272030 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:21.924288 9234 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.331666 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.333586 9234 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:22.273269 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 328368us
I20260806 04:08:22.335516 4882 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:22.336529 4882 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:22.337819 4882 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:22.276360 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:22.278441 9307 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.437s user 0.001s sys 0.000s
I20260806 04:08:22.275717 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.339041 4920 raft_consensus.cc:2362] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7. Re-sending same reply.
I20260806 04:08:22.282519 5862 raft_consensus.cc:1226] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:22.184500 9196 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:22.293445 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.214638 9278 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:22.300931 5029 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:08:22.302686 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.302826 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:21.941699 9245 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:22.232658 8975 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:22.356532 4924 raft_consensus.cc:2354] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:08:22.330302 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.357209 5034 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:22.330643 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.331497 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.028721 8635 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.339401 4926 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
I20260806 04:08:22.348476 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.330547 5038 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 3.
I20260806 04:08:22.290807 9146 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:22.292876 9130 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:22.354898 5026 raft_consensus.cc:2354] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:22.329584 8780 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:22.359381 5035 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:22.287281 8955 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:22.358734 5037 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:22.292487 9002 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:22.370536 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.369160 5036 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:22.370596 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:22.350540 9267 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260806 04:08:22.376560 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:22.334539 9315 log.cc:924] Time spent T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.079s user 0.002s sys 0.001s
I20260806 04:08:22.384064 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:22.394790 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.395299 5857 raft_consensus.cc:2379] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:22.387288 4920 raft_consensus.cc:2354] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:08:22.137422 9153 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.355050 8995 raft_consensus.cc:2781] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:22.395484 5034 raft_consensus.cc:2362] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3. Re-sending same reply.
I20260806 04:08:22.304659 8641 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.394016 9002 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:22.384825 9284 log.cc:924] Time spent T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260806 04:08:22.415095 4919 raft_consensus.cc:1261] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 04:08:22.386233 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.391522 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.395165 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.347085 9001 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:22.400346 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.355319 8797 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:22.420878 5038 raft_consensus.cc:2379] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 3: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:22.377426 9293 raft_consensus.cc:686] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:22.267760 8984 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.433116 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:22.406395 8520 consensus_meta.cc:287] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.899s user 0.013s sys 0.000s
I20260806 04:08:22.422067 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.346360 9234 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260806 04:08:22.396229 9312 log.cc:924] Time spent T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.493s user 0.001s sys 0.000s
W20260806 04:08:22.425590 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 77576.6us
W20260806 04:08:22.426193 8980 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.013s sys 0.012s
I20260806 04:08:22.404192 9173 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:22.342727 9322 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.428673 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.422801 8995 raft_consensus.cc:686] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:22.279603 8863 consensus_queue.cc:237] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.371585 9278 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:22.467417 9278 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.468374 9322 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:22.433328 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.469883 9278 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:22.470342 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.434827 4926 raft_consensus.cc:1226] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 04:08:22.397130 9258 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 04:08:22.476601 4814 consensus_meta.cc:287] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.729s user 0.008s sys 0.002s
I20260806 04:08:22.477378 4814 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:08:22.439122 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.481997 5038 raft_consensus.cc:2362] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10. Re-sending same reply.
I20260806 04:08:22.442623 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.482631 5029 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:22.483093 4919 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:22.370770 8975 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.451622 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.455561 5037 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:22.405441 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.486229 5030 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 9.
I20260806 04:08:22.487980 4920 raft_consensus.cc:2396] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260806 04:08:22.454859 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.310915 9243 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.398669 8955 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:22.478428 9330 log.cc:924] Time spent T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:08:22.436051 9343 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:22.479468 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.366462 9196 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.389813 9130 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:22.448148 8980 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s user 0.013s sys 0.012s
I20260806 04:08:22.434461 5034 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:22.444128 9273 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:22.478897 9305 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.548s user 0.001s sys 0.000s
I20260806 04:08:22.444077 9283 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:22.495191 5030 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: []
W20260806 04:08:22.459867 8997 consensus_meta.cc:287] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.719s user 0.015s sys 0.000s
I20260806 04:08:22.428227 9001 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.433074 8797 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.507041 5026 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:08:22.519829 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.476656 8750 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:22.521304 4920 raft_consensus.cc:1226] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:08:22.441190 8520 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:22.492595 9291 consensus_meta.cc:287] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.713s user 0.012s sys 0.000s
I20260806 04:08:22.450999 8873 consensus_queue.cc:1046] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
W20260806 04:08:22.541172 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 193424us
I20260806 04:08:22.547873 4919 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:22.478562 8995 consensus_queue.cc:237] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.529136 9278 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:22.596143 4810 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.007s sys 0.010s
I20260806 04:08:22.584928 4931 raft_consensus.cc:2396] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 04:08:22.590243 9196 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:22.585776 5862 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:08:22.593515 9001 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:22.587352 8641 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:22.589936 8981 consensus_meta.cc:287] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.798s user 0.008s sys 0.001s
I20260806 04:08:22.568141 9293 consensus_queue.cc:237] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.591552 9207 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:22.591747 8797 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:22.607914 9324 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.221s user 0.001s sys 0.000s
I20260806 04:08:22.594172 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.625360 5030 raft_consensus.cc:2362] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10. Re-sending same reply.
I20260806 04:08:22.594940 9303 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:22.583441 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:22.599599 4810 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.008s sys 0.010s
I20260806 04:08:22.635246 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2377) took 1199 ms. Trace:
I20260806 04:08:22.606962 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:22.615583 5035 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:22.593582 8520 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:22.626021 4928 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:22.635843 4810 rpcz_store.cc:276] 0806 04:08:21.436159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.518993 (+ 82834us) service_pool.cc:227] Handling call
0806 04:08:21.519367 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:21.519616 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:08:21.794862 (+275246us) log.cc:842] Serialized 14 byte log entry
0806 04:08:22.634704 (+839842us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:22.634800 (+ 96us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:22.634882 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:22.634979 (+ 97us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:22.635162 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":383604,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":4484,"raft.queue_time_us":264499,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":34200,"thread_start_us":1113357,"threads_started":3,"wal-append.queue_time_us":560177}
W20260806 04:08:22.617199 9331 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20260806 04:08:22.521726 9343 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.599758 9278 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.683094 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.698196 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.707329 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.699092 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:22.708673 5030 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
W20260806 04:08:22.699415 9334 log.cc:924] Time spent T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20260806 04:08:22.702907 9335 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20260806 04:08:22.703788 8955 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:22.705580 9285 raft_consensus.cc:3032] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:22.716603 4920 raft_consensus.cc:2379] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 9: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:22.634125 9303 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:22.616526 8981 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.892974 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:22.831789 9273 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:22.824242 8936 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:22.636701 9290 consensus_queue.cc:1046] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.138s
I20260806 04:08:22.837193 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.922518 5034 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 7: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
W20260806 04:08:22.924104 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.014s sys 0.009s
W20260806 04:08:22.924456 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.014s sys 0.009s
W20260806 04:08:22.924736 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 577080us
I20260806 04:08:22.839030 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:22.925457 5030 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:22.925369 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:22.933811 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 494) took 1264 ms. Trace:
I20260806 04:08:22.880201 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.822525 9243 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:22.912591 4769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.050s sys 0.099s
W20260806 04:08:22.964361 4769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.051s sys 0.099s
W20260806 04:08:22.823062 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 495122us
I20260806 04:08:22.921025 4920 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:08:22.825435 8180 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:22.973032 8180 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:22.975437 8180 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:22.837647 9285 raft_consensus.cc:729] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:22.874424 9303 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:22.925868 9276 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.291s user 0.009s sys 0.007s
I20260806 04:08:23.001161 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:23.015062 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 45849us
I20260806 04:08:22.961544 4808 rpcz_store.cc:276] 0806 04:08:21.669252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.882564 (+213312us) service_pool.cc:227] Handling call
0806 04:08:21.883772 (+ 1208us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:21.884284 (+ 512us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:21.884327 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:21.907331 (+ 23004us) log.cc:842] Serialized 26 byte log entry
0806 04:08:22.289541 (+382210us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:22.289599 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:22.289697 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:22.933488 (+643791us) raft_consensus.cc:1671] finished
0806 04:08:22.933519 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:22.933727 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19117,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7374,"thread_start_us":366814,"threads_started":1,"wal-append.queue_time_us":583356}
W20260806 04:08:22.821156 5812 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.009s sys 0.022s
I20260806 04:08:22.970371 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:22.972510 9350 log.cc:924] Time spent T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20260806 04:08:22.980037 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.030544 4813 raft_consensus.cc:2354] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:08:23.032546 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:22.938949 9130 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:22.981319 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:22.906831 8635 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:22.966286 8984 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:23.018107 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.017624 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:23.027460 5812 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.786s user 0.010s sys 0.022s
I20260806 04:08:23.027819 5034 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:23.036739 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:23.064755 4817 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 9
W20260806 04:08:23.064767 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 823859us
I20260806 04:08:23.038094 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:23.013998 9276 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.384s user 0.009s sys 0.008s
W20260806 04:08:23.019534 9242 consensus_meta.cc:287] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.664s user 0.012s sys 0.005s
I20260806 04:08:23.001426 9173 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.039928 4816 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:22.999511 5030 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:22.941044 8936 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:22.922930 9273 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.078361 4920 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:23.084596 5859 raft_consensus.cc:1226] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:23.093999 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.095255 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.096801 4818 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:23.103083 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.035204 9290 consensus_queue.cc:1035] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260806 04:08:23.102285 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:23.108016 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.117146 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 498) took 1541 ms. Trace:
I20260806 04:08:23.048662 9156 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260806 04:08:23.136721 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.140065 4818 raft_consensus.cc:2396] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:23.141213 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.092660 9293 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.142530 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.142935 9293 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.143887 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.145521 9293 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:23.117650 4927 rpcz_store.cc:276] 0806 04:08:21.576236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.585951 (+ 9715us) service_pool.cc:227] Handling call
0806 04:08:21.586389 (+ 438us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:21.864204 (+277815us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:21.864237 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:21.864278 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:21.864346 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:23.116880 (+1252534us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400355cd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:23.116938 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:23.117083 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":638408448}
I20260806 04:08:23.122731 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:23.147753 9343 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:23.148185 4920 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:23.097131 9173 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.149667 4814 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:23.150635 9173 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:23.123943 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.124173 8936 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:23.152199 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.125948 8981 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:23.127259 9351 log.cc:924] Time spent T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.287s user 0.000s sys 0.001s
W20260806 04:08:23.128054 9349 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.205s user 0.000s sys 0.001s
I20260806 04:08:23.093276 9242 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.129105 9278 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:23.130219 9363 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.206s user 0.000s sys 0.000s
I20260806 04:08:23.125834 9273 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.152881 4805 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:23.173158 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.173158 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.157739 5034 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:23.166343 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.176929 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:23.168530 9245 consensus_meta.cc:287] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.784s user 0.006s sys 0.008s
W20260806 04:08:23.172915 9002 consensus_meta.cc:287] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.730s user 0.002s sys 0.013s
I20260806 04:08:23.189106 4876 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.187745 4818 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:08:23.194063 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.194847 5026 raft_consensus.cc:2362] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4. Re-sending same reply.
I20260806 04:08:23.197782 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.199144 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.201236 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.204774 4996 leader_election.cc:398] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:08:23.212241 4996 leader_election.cc:401] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260806 04:08:23.179121 9139 consensus_meta.cc:287] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.856s user 0.011s sys 0.002s
I20260806 04:08:23.208716 4876 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:23.161433 9195 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260806 04:08:23.209949 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:23.214454 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:23.217949 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.226572 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.227658 4808 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:23.223568 4818 raft_consensus.cc:1261] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:23.229991 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.230903 4818 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:23.227296 5026 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:23.194674 9245 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.238282 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.240264 9245 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.240437 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.195802 9243 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20260806 04:08:23.241261 4818 raft_consensus.cc:2379] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:08:23.242044 9275 consensus_meta.cc:287] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.373s user 0.013s sys 0.000s
I20260806 04:08:23.242583 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.242862 9243 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:08:23.196238 9002 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.243499 4818 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:23.245146 9002 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:23.219744 8862 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:23.248034 4768 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:23.250372 4768 leader_election.cc:398] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
W20260806 04:08:23.226567 8532 consensus_meta.cc:287] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.269s user 0.011s sys 0.006s
I20260806 04:08:23.251634 5034 raft_consensus.cc:2396] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:23.251245 8532 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.252271 9002 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:23.252727 9002 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:23.253131 9002 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:23.254041 8532 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.245465 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.257156 4768 leader_election.cc:401] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:23.260912 4818 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
W20260806 04:08:23.263638 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 53807.3us
I20260806 04:08:23.265169 5816 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.268596 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2150) took 1339 ms. Trace:
I20260806 04:08:23.273454 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.274065 4817 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:23.269104 5032 rpcz_store.cc:276] 0806 04:08:21.929036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.930884 (+ 1848us) service_pool.cc:227] Handling call
0806 04:08:21.946067 (+ 15183us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:08:21.951762 (+ 5695us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:21.951809 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 04:08:22.045777 (+ 93968us) log.cc:842] Serialized 78 byte log entry
0806 04:08:22.416331 (+370554us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:22.527045 (+110714us) log.cc:842] Serialized 14 byte log entry
0806 04:08:22.530585 (+ 3540us) log.cc:842] Serialized 14 byte log entry
0806 04:08:22.530854 (+ 269us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:22.530974 (+ 120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:22.704916 (+173942us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b18003e52a0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32764 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:23.268226 (+563310us) raft_consensus.cc:1671] finished
0806 04:08:23.268262 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:23.268457 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":174100,"prepare.queue_time_us":224906,"prepare.run_cpu_time_us":4201,"prepare.run_wall_time_us":47878,"raft.queue_time_us":10523,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":88933,"spinlock_wait_cycles":71761920,"thread_start_us":226921,"threads_started":2,"wal-append.queue_time_us":531157}
I20260806 04:08:23.275771 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:23.276726 4817 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:23.281069 5862 raft_consensus.cc:1261] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:08:23.266942 5816 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:23.218904 8180 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:23.314989 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.315943 5857 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
I20260806 04:08:23.292268 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.320621 5026 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:23.325152 4808 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
I20260806 04:08:23.243536 9243 raft_consensus.cc:686] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:23.313864 9363 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260806 04:08:23.267289 9156 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:23.348544 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.266757 9157 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:23.352377 4920 raft_consensus.cc:1226] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:23.352583 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.282796 8532 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:08:23.353300 5026 raft_consensus.cc:3037] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:23.355510 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 82150.1us
I20260806 04:08:23.356735 4996 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:23.358011 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.282380 9383 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:23.236088 8635 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.098s
I20260806 04:08:23.329598 9278 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
W20260806 04:08:23.348402 9169 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.070s user 0.017s sys 0.025s
I20260806 04:08:23.349412 9028 raft_consensus.cc:3032] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:23.336325 9389 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.350606 9130 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.259479 9139 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:23.379768 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 129528us
I20260806 04:08:23.381070 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.365414 9207 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:23.369778 9169 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.018s sys 0.025s
I20260806 04:08:23.370899 8532 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:08:23.382169 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 114333us
I20260806 04:08:23.363265 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:23.387673 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 179022us
I20260806 04:08:23.401803 4876 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:23.391203 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 183865us
I20260806 04:08:23.404598 4882 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:23.380693 9383 raft_consensus.cc:686] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:23.362469 9156 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.371944 9028 raft_consensus.cc:729] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:23.397053 5026 raft_consensus.cc:2454] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:23.424574 9028 consensus_queue.cc:260] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.425725 9028 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:23.425778 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.427377 5026 raft_consensus.cc:2362] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8. Re-sending same reply.
I20260806 04:08:23.411306 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters:
I20260806 04:08:23.411653 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.389132 9242 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:23.430576 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.403272 9243 consensus_queue.cc:237] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.431938 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:23.432385 9369 log.cc:924] Time spent T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260806 04:08:23.431773 9383 consensus_queue.cc:237] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.422931 9355 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.433772 9290 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:23.383289 9130 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.423447 9156 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.428959 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:23.420938 5812 connection.cc:232] RPC call timeout handler was delayed by -1.1654029691126198s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
I20260806 04:08:23.386861 9207 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:23.399904 8532 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:23.465281 8532 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:23.387140 9275 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.386539 9370 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:23.476799 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 65569.8us
I20260806 04:08:23.477834 4876 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.434317 4768 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:08:23.446985 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 65477.7us
I20260806 04:08:23.371491 8921 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
W20260806 04:08:23.451761 5812 leader_election.cc:334] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.723s (SENT)
I20260806 04:08:23.433550 9355 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:23.463567 9386 log.cc:924] Time spent T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260806 04:08:23.464634 9207 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:23.478531 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 104843us
I20260806 04:08:23.451169 9291 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:08:23.490545 4876 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.473901 5026 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:23.492304 4996 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:23.496790 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 92284.8us
W20260806 04:08:23.498246 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 231393us
I20260806 04:08:23.498363 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.501065 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.508948 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:23.513276 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:23.514135 4925 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:23.517613 4927 raft_consensus.cc:1261] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:23.387424 9389 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.520341 4992 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.493736 9278 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:23.523823 4927 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:23.476107 9156 consensus_queue.cc:1046] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 04:08:23.526739 4876 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:23.524955 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.504889 8995 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:23.564847 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:23.482398 9002 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.569936 4927 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: []
W20260806 04:08:23.565917 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 166615us
I20260806 04:08:23.536517 9291 raft_consensus.cc:686] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:23.566887 9397 raft_consensus.cc:2741] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 04:08:23.488816 9275 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.600126 4805 raft_consensus.cc:1226] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:23.525092 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.527132 4882 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.528148 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:23.576798 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 84579.9us
I20260806 04:08:23.577692 4768 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:23.605399 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 78389.7us
I20260806 04:08:23.557255 8936 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:23.580523 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:23.609695 4927 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:23.512653 8862 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5.
I20260806 04:08:23.587373 9139 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.560606 9273 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:23.606503 4882 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.622625 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.608211 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.613651 4996 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:23.619232 4927 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:23.568807 8995 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:23.493911 9370 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.629509 4882 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:23.633034 9291 consensus_queue.cc:237] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.536060 9417 consensus_queue.cc:1046] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
W20260806 04:08:23.624975 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 61082.6us
I20260806 04:08:23.625766 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:23.627590 9402 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:23.624321 5026 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:23.643935 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.623383 5032 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:23.562167 9278 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:23.637254 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 59667us
I20260806 04:08:23.638360 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:23.649770 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1629) took 1070 ms. Trace:
I20260806 04:08:23.653342 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:23.656107 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:23.657168 4927 raft_consensus.cc:3037] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:23.662771 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:23.650261 4919 rpcz_store.cc:276] 0806 04:08:22.579349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.643236 (+ 63887us) service_pool.cc:227] Handling call
0806 04:08:22.643815 (+ 579us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:22.688184 (+ 44369us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.688217 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:22.846875 (+158658us) log.cc:842] Serialized 26 byte log entry
0806 04:08:23.190962 (+344087us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:23.207395 (+ 16433us) log.cc:842] Serialized 14 byte log entry
0806 04:08:23.390925 (+183530us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:23.391035 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:23.649515 (+258480us) raft_consensus.cc:1671] finished
0806 04:08:23.649548 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:23.649701 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43798,"prepare.queue_time_us":23945,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":7902,"raft.queue_time_us":87531,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":246,"thread_start_us":410407,"threads_started":2,"wal-append.queue_time_us":415860}
I20260806 04:08:23.631939 8995 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:23.675956 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.693572 4808 raft_consensus.cc:3037] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:23.632272 9204 raft_consensus.cc:3032] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:23.628307 9317 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:23.628710 8936 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:23.630306 9273 raft_consensus.cc:686] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:23.637578 9247 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:08:20.224651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.68:38873 (local address 127.0.0.1:43100)
0806 04:08:21.187702 (+963051us) negotiation.cc:107] Waiting for socket to connect
0806 04:08:21.187817 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0806 04:08:21.188507 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:08:21.455397 (+266890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:08:21.455440 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:08:21.455595 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:08:21.456622 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:08:21.494305 (+ 37683us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:08:21.717952 (+223647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:08:21.756469 (+ 38517us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:08:23.133664 (+1377195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:08:23.133729 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:08:23.136496 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:08:23.136534 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:08:23.137088 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:08:23.205092 (+ 68004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:08:23.209020 (+ 3928us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:08:23.217482 (+ 8462us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:08:23.254825 (+ 37343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:08:23.255756 (+ 931us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.248.68:38873: sent 0 of 16 requested bytes
Metrics: {"client-negotiator.queue_time_us":907287,"thread_start_us":496593,"threads_started":1}
I20260806 04:08:23.635916 9169 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:23.634397 9242 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:23.637761 8862 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:23.702070 4768 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:23.667150 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:08:20.406454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.2.248.68:38873)
0806 04:08:21.339194 (+932740us) server_negotiation.cc:206] Beginning negotiation
0806 04:08:21.339253 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0806 04:08:21.387861 (+ 48608us) server_negotiation.cc:407] Connection header received
0806 04:08:21.414009 (+ 26148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:08:21.420846 (+ 6837us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:08:21.454978 (+ 34132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:08:21.455325 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:08:21.524182 (+ 68857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:08:21.622433 (+ 98251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:08:23.134078 (+1511645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:08:23.135475 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:08:23.135836 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:08:23.256387 (+120551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43100: BlockingRecv error: recv got EOF from 127.0.0.1:43100 (error 108)
Metrics: {"mutex_wait_us":4247,"server-negotiator.queue_time_us":815189,"thread_start_us":800914,"threads_started":1}
I20260806 04:08:23.668738 9407 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:23.662866 9430 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:23.748585 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 237094us
I20260806 04:08:23.752291 4919 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
W20260806 04:08:23.753052 9416 log.cc:924] Time spent T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260806 04:08:23.761304 9278 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:23.821380 4992 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.705649 9204 raft_consensus.cc:729] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:23.751977 8520 raft_consensus.cc:3032] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:23.763459 3041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:08:23.766829 9360 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:23.767139 9139 raft_consensus.cc:2781] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 04:08:23.845311 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 144702us
I20260806 04:08:23.698714 9429 consensus_queue.cc:1035] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.159s
I20260806 04:08:23.801265 9407 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:23.661290 8180 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
W20260806 04:08:23.741753 9405 log.cc:924] Time spent T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
W20260806 04:08:23.800366 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 153304us
I20260806 04:08:23.740801 4927 raft_consensus.cc:2454] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
W20260806 04:08:23.826524 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 309578us
W20260806 04:08:23.828457 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 133851us
I20260806 04:08:23.730109 9434 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:23.840348 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:08:23.771329 9389 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.783716 8862 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:23.934283 8862 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:23.934993 9425 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:23.777599 4808 raft_consensus.cc:2454] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
W20260806 04:08:23.940497 9303 consensus_meta.cc:287] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.928s user 0.013s sys 0.000s
I20260806 04:08:23.940977 9303 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.891472 8936 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.946061 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 339) took 1019 ms. Trace:
I20260806 04:08:23.895188 9002 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:23.827816 9355 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.807926 9169 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:23.831557 8520 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:23.909673 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:23.840263 9360 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:23.967691 9360 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:23.929872 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 129703us
I20260806 04:08:23.946463 5034 rpcz_store.cc:276] 0806 04:08:22.927044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.176596 (+249552us) service_pool.cc:227] Handling call
0806 04:08:23.939453 (+762857us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b54003205d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:23.939600 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":169141120}
I20260806 04:08:23.970875 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.971128 4992 leader_election.cc:398] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 3 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4.
I20260806 04:08:23.971637 5034 raft_consensus.cc:2379] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 7: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:23.972067 4992 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:23.972484 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.973641 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:23.974920 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:23.975037 8520 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.976420 8520 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:23.974640 4813 raft_consensus.cc:3032] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:23.980407 4813 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:23.981498 4813 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:23.953472 9207 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:23.957789 5812 leader_election.cc:334] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.778s (SENT)
I20260806 04:08:23.983002 5812 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:23.901364 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 267473us
I20260806 04:08:23.984418 9207 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:23.982553 4813 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:23.986145 9285 consensus_queue.cc:260] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.987602 9285 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:23.987778 5812 leader_election.cc:334] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 1.894s (SENT)
W20260806 04:08:23.989070 5812 leader_election.cc:334] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.103s (SENT)
I20260806 04:08:23.909740 9407 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:24.007616 9432 consensus_queue.cc:1035] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:24.010443 5812 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:24.011499 5812 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:23.903174 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:24.014246 9321 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:08:24.015403 5854 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 10: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:24.015341 9321 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:08:24.016284 9321 raft_consensus.cc:2781] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:24.016712 8952 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:08:24.017203 9422 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:24.017673 9422 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:23.980294 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:23.950420 8532 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:23.903301 9404 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:23.909583 9139 raft_consensus.cc:686] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [term 8 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:24.019788 9157 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:23.986670 9435 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:24.000931 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 326) took 1406 ms. Trace:
I20260806 04:08:24.019968 9139 consensus_queue.cc:237] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.020880 9169 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:24.021205 5862 raft_consensus.cc:1226] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:08:24.021907 9321 raft_consensus.cc:686] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:23.962026 4925 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:23.820971 9430 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.016920 9414 raft_consensus.cc:3032] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:24.023554 9414 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.023694 9258 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:24.020736 5035 rpcz_store.cc:276] 0806 04:08:22.594959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.671484 (+ 76525us) service_pool.cc:227] Handling call
0806 04:08:22.709508 (+ 38024us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:22.709954 (+ 446us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.709986 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:22.824365 (+114379us) log.cc:842] Serialized 26 byte log entry
0806 04:08:23.228550 (+404185us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:23.969826 (+741276us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.000591 (+ 30765us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.000697 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:24.000755 (+ 58us) raft_consensus.cc:1671] finished
0806 04:08:24.000765 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.000879 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":771933,"prepare.queue_time_us":749277,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":117,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":50035,"thread_start_us":292589,"threads_started":1,"wal-append.queue_time_us":515075}
I20260806 04:08:24.024014 9430 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.024204 9258 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:24.024699 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.024520 9414 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.026117 9414 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:24.026286 5035 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:23.948294 5853 raft_consensus.cc:1226] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:08:24.023365 9321 consensus_queue.cc:237] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.027472 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 327) took 1432 ms. Trace:
I20260806 04:08:24.021479 9169 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.027845 5035 rpcz_store.cc:276] 0806 04:08:22.595420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.025756 (+1430336us) service_pool.cc:227] Handling call
0806 04:08:24.026111 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:24.027233 (+ 1122us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.027420 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:24.028559 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 57211us
I20260806 04:08:24.022040 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:24.032653 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 60655.8us
I20260806 04:08:24.033470 4992 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.035315 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.036376 4813 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:08:23.994072 9441 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260806 04:08:24.038667 5035 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:24.037364 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:24.020365 8532 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.018566 9422 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.041695 9422 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:24.034543 9389 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:23.903880 9273 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.039944 9370 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.039836 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2155) took 2101 ms. Trace:
I20260806 04:08:24.045130 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 330) took 1348 ms. Trace:
I20260806 04:08:24.045079 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:24.045760 5035 rpcz_store.cc:276] 0806 04:08:22.696565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.028627 (+1332062us) service_pool.cc:227] Handling call
0806 04:08:24.038420 (+ 9793us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:24.039196 (+ 776us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.045075 (+ 5879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:24.046809 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.045917 5038 rpcz_store.cc:276] 0806 04:08:21.938294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.484565 (+546271us) service_pool.cc:227] Handling call
0806 04:08:22.485008 (+ 443us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:22.485399 (+ 391us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.485430 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:22.488785 (+ 3355us) log.cc:842] Serialized 52 byte log entry
0806 04:08:22.760333 (+271548us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:23.059511 (+299178us) log.cc:842] Serialized 14 byte log entry
0806 04:08:23.937805 (+878294us) spinlock_profiling.cc:243] Waited 7.53 ms on lock 0x7b4800260358. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23de8017f1 00007f23de7f88e7 00007f23de7f877a 00007f23de8ec95f 00007f23de8ec627 00007f23de900b2c 00007f23de900ad8 00007f23de900a58 00007f23de900a13 00007f23de8ffcbc 00007f23dd881be6
0806 04:08:23.966514 (+ 28709us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.034331 (+ 67817us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.034409 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:24.039608 (+ 5199us) raft_consensus.cc:1671] finished
0806 04:08:24.039635 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.039767 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1989164,"prepare.queue_time_us":1013602,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":126611,"raft.run_cpu_time_us":2689,"raft.run_wall_time_us":845459,"spinlock_wait_cycles":17328000,"thread_start_us":387061,"threads_started":2,"wal-append.queue_time_us":597215}
I20260806 04:08:24.050709 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.051542 4813 raft_consensus.cc:2396] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 04:08:24.056416 9275 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.057984 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:24.062527 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.063176 5038 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:24.066749 4992 leader_election.cc:398] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:08:24.067088 4992 leader_election.cc:401] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:24.080704 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.079224 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.082368 5035 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:24.088315 4768 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:24.089920 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:24.077454 9028 consensus_meta.cc:287] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.635s user 0.011s sys 0.002s
I20260806 04:08:24.093642 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.094218 5035 raft_consensus.cc:2454] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:24.090401 5038 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 9.
I20260806 04:08:24.101058 4813 raft_consensus.cc:1226] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:24.106686 4995 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.103883 8981 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:24.079954 9207 raft_consensus.cc:2741] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 04:08:24.084390 9245 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.110765 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:24.084014 9303 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.080181 9169 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.118417 5035 raft_consensus.cc:1226] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 5. Ops: []
W20260806 04:08:24.125784 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 59190.5us
I20260806 04:08:24.079655 9316 consensus_queue.cc:1035] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260806 04:08:24.130479 4992 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.132684 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:24.133329 4768 leader_election.cc:398] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:24.134025 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 331) took 1431 ms. Trace:
W20260806 04:08:24.152576 5812 leader_election.cc:334] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.217s (SENT)
I20260806 04:08:24.108613 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:24.108603 9444 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 2. Current term is 3.
I20260806 04:08:24.157114 5035 rpcz_store.cc:276] 0806 04:08:22.697055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.094803 (+1397748us) service_pool.cc:227] Handling call
0806 04:08:24.117814 (+ 23011us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:24.128508 (+ 10694us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.128764 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:24.127235 9430 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:08:24.129354 9391 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.947s user 0.006s sys 0.000s
I20260806 04:08:24.113719 9410 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:24.167344 9410 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.172103 9410 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:24.172775 9428 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.234s user 0.000s sys 0.000s
I20260806 04:08:24.143697 4992 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.132169 8981 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:24.168076 8812 raft_consensus.cc:3032] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 04:08:24.175086 9439 log.cc:924] Time spent T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.232s user 0.002s sys 0.000s
I20260806 04:08:24.175205 8812 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.157120 5812 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.176558 8812 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.177733 8812 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:24.178540 9410 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:24.174019 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:24.175170 8981 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.180147 8981 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:24.180794 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.181480 4818 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:24.182286 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.184160 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:24.256281 4818 raft_consensus.cc:2396] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260806 04:08:24.256338 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 72290.9us
I20260806 04:08:24.257566 4768 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.180166 9169 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:24.192463 8981 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.259001 9169 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.259670 9169 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:24.260658 8981 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.261301 8981 raft_consensus.cc:3032] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:08:24.261687 8981 raft_consensus.cc:729] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:24.129952 9245 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.262900 9303 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:24.264292 4813 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:24.208683 9450 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260806 04:08:24.157761 4996 leader_election.cc:334] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865): Timed out: RequestConsensusVote RPC to 127.2.248.65:43865 timed out after 1.729s (SENT)
W20260806 04:08:24.234344 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 90788us
I20260806 04:08:24.266340 4768 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.252902 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.262683 8981 consensus_queue.cc:260] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.269985 9316 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:24.120572 9389 consensus_queue.cc:1046] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 04:08:24.267176 4992 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.270992 9389 raft_consensus.cc:3032] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:24.271386 9389 raft_consensus.cc:729] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:24.272161 4992 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.272279 9389 consensus_queue.cc:260] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.273595 9389 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:24.273891 5812 leader_election.cc:334] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701): Timed out: RequestConsensusVote RPC to 127.2.248.67:43701 timed out after 2.597s (SENT)
W20260806 04:08:24.266569 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 108845us
I20260806 04:08:24.273622 4768 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:24.237313 9207 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:24.270345 8981 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:08:24.279534 9316 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:24.280462 8995 consensus_meta.cc:287] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.553s user 0.011s sys 0.000s
I20260806 04:08:24.280934 8995 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.276304 9422 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:24.283001 9422 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:24.283437 9422 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:24.276809 9433 consensus_queue.cc:1159] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "22fe557f852d4415a96fd683f970cc24" responder_term: 6 status { last_received { term: 3 index: 2 } last_committed_idx: 1 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 04:08:24.290045 8975 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.290508 8981 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:08:24.291265 9407 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.291925 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 503) took 1076 ms. Trace:
I20260806 04:08:24.265100 9245 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:24.291934 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:24.333338 9360 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:24.422253 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 130391us
I20260806 04:08:24.423635 9360 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.424455 8981 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:24.424999 8981 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.425511 8981 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:24.347020 5030 raft_consensus.cc:2396] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:24.347254 9355 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:24.347481 9429 consensus_queue.cc:1046] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260806 04:08:24.429670 9429 raft_consensus.cc:3032] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:24.430087 9429 raft_consensus.cc:729] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
W20260806 04:08:24.347570 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 73973.1us
I20260806 04:08:24.432945 4992 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:24.350447 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 74372us
I20260806 04:08:24.435328 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.437513 4992 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:24.367375 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 78192.1us
I20260806 04:08:24.382484 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:24.357450 4805 rpcz_store.cc:276] 0806 04:08:23.215270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.408570 (+193300us) service_pool.cc:227] Handling call
0806 04:08:23.466889 (+ 58319us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:23.607088 (+140199us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.291648 (+684560us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400230d54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:24.291858 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305773312}
I20260806 04:08:24.447953 8975 consensus_queue.cc:1046] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:08:24.449491 9303 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:24.449889 9303 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:24.457442 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:24.458896 4810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1483) took 1480 ms (client timeout 1729 ms). Trace:
W20260806 04:08:24.459396 4810 rpcz_store.cc:269] 0806 04:08:22.978623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.013592 (+ 34969us) service_pool.cc:227] Handling call
0806 04:08:24.457207 (+1443615us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:24.458831 (+ 1624us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8877824}
I20260806 04:08:24.290165 9316 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:24.466606 4992 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.467026 4805 raft_consensus.cc:1261] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:24.347527 8955 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:24.376748 9430 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:24.379278 9407 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.460100 9355 raft_consensus.cc:2726] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:24.395435 8995 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.479785 4992 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.450984 9303 consensus_queue.cc:260] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.481253 9316 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:24.451539 9429 consensus_queue.cc:260] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.478328 9285 consensus_queue.cc:1035] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260806 04:08:24.486171 9275 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:08:24.484133 4992 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.367559 9450 consensus_queue.cc:1046] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260806 04:08:24.469851 9156 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:24.496053 9316 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.468144 9436 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:08:24.499327 4992 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.460661 8532 consensus_queue.cc:1046] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260806 04:08:24.329165 8862 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.509581 8862 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.510533 5026 raft_consensus.cc:2354] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9.
I20260806 04:08:24.482087 8975 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.511426 8975 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.513633 8975 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.515113 8532 raft_consensus.cc:3032] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:24.515666 8532 raft_consensus.cc:729] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.516851 8532 consensus_queue.cc:260] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.406440 9422 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.478735 8520 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:24.363839 9258 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.502092 9139 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:24.490912 8955 raft_consensus.cc:686] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:24.527067 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:24.528177 5862 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:24.520614 8532 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:24.540632 9316 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:24.542936 9245 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:24.543780 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:24.543159 9422 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.545015 8975 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.547240 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 332) took 1849 ms. Trace:
I20260806 04:08:24.568523 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2378) took 3132 ms. Trace:
I20260806 04:08:24.553610 5812 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.569028 4804 rpcz_store.cc:276] 0806 04:08:21.436476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.527762 (+ 91286us) service_pool.cc:227] Handling call
0806 04:08:21.528140 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:21.528441 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 11
0806 04:08:21.796883 (+268442us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.567887 (+2771004us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b1800203860. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d95e982c 00007f23d6a149dc 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e157b
0806 04:08:24.568108 (+ 221us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:24.568162 (+ 54us) raft_consensus.cc:1631] Marking committed up to 11
0806 04:08:24.568229 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.568324 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.568462 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":390150,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"spinlock_wait_cycles":7627648,"thread_start_us":418402,"threads_started":2,"wal-append.queue_time_us":553375,"wal-append.run_cpu_time_us":2818,"wal-append.run_wall_time_us":1332668}
I20260806 04:08:24.574729 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:24.575572 4804 raft_consensus.cc:3037] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 9
I20260806 04:08:24.528295 8981 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.568447 5038 rpcz_store.cc:276] 0806 04:08:22.697498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.095414 (+1397916us) service_pool.cc:227] Handling call
0806 04:08:24.095831 (+ 417us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:24.118963 (+ 23132us) raft_consensus.cc:1520] Early marking committed up to index 9
0806 04:08:24.133777 (+ 14814us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.429304 (+295527us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:24.429918 (+ 614us) log.cc:842] Serialized 26 byte log entry
0806 04:08:24.430313 (+ 395us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:08:24.430349 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.430427 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:24.546944 (+116517us) raft_consensus.cc:1671] finished
0806 04:08:24.546973 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.547156 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12078,"prepare.queue_time_us":14616,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"raft.queue_time_us":12385,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":198,"thread_start_us":240125,"threads_started":1,"wal-append.queue_time_us":267712}
I20260806 04:08:24.591032 8955 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.571502 8997 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.594746 9245 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.530912 9169 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.604595 5862 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 9.
I20260806 04:08:24.591056 9316 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.606736 4804 raft_consensus.cc:2454] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 9.
I20260806 04:08:24.609834 4804 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260806 04:08:24.609599 4995 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:24.606933 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33548 (request call id 2391) took 1513 ms. Trace:
I20260806 04:08:24.611429 4814 rpcz_store.cc:276] 0806 04:08:23.093405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.093675 (+ 270us) service_pool.cc:227] Handling call
0806 04:08:24.606704 (+1513029us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b5400021554. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:24.606857 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126070016}
I20260806 04:08:24.613531 4818 raft_consensus.cc:2379] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:24.596611 9414 raft_consensus.cc:2726] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:24.617832 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:24.618746 4817 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:24.620440 5038 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: [5.6-8.8]
I20260806 04:08:24.574800 8975 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.615271 9275 raft_consensus.cc:2781] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:24.606842 9422 consensus_queue.cc:1046] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260806 04:08:24.622547 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 334) took 1924 ms. Trace:
I20260806 04:08:24.635929 5038 rpcz_store.cc:276] 0806 04:08:22.698231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.619592 (+1921361us) service_pool.cc:227] Handling call
0806 04:08:24.620083 (+ 491us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:08:24.621853 (+ 1770us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.622466 (+ 613us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:24.609855 8997 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.611950 9245 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.611161 9316 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.649988 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 345) took 1568 ms. Trace:
I20260806 04:08:24.655619 5032 rpcz_store.cc:276] 0806 04:08:23.081602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.431608 (+350006us) service_pool.cc:227] Handling call
0806 04:08:24.649782 (+1218174us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b540008e654. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:24.649928 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":438380928}
I20260806 04:08:24.651253 5035 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 6. Ops: []
I20260806 04:08:24.648962 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.658414 5038 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:24.663460 4817 raft_consensus.cc:2454] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:24.663631 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.664788 5038 raft_consensus.cc:2354] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 5.
I20260806 04:08:24.636322 9207 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:24.665136 5816 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.674947 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.678061 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.680058 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.683363 5038 raft_consensus.cc:2354] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:24.649535 9275 raft_consensus.cc:686] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [term 9 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:24.688123 8981 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.691236 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.681618 9422 raft_consensus.cc:2781] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:24.700589 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.713001 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.714258 5038 raft_consensus.cc:2354] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:24.717288 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.719617 4805 raft_consensus.cc:1226] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
W20260806 04:08:24.727320 9475 log.cc:924] Time spent T 00000000000000000000000000000000 P 7a82866733fa43d09abd13eedc5ee6df: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:08:24.727891 5853 raft_consensus.cc:1226] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:08:24.677407 9207 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.733261 9414 consensus_queue.cc:1046] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:08:24.734186 9414 raft_consensus.cc:3032] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:24.709537 9422 raft_consensus.cc:686] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.734544 9414 raft_consensus.cc:729] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.734838 9207 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:24.735330 9422 consensus_queue.cc:237] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.736343 9169 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.736985 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1628) took 2181 ms. Trace:
I20260806 04:08:24.732416 5032 raft_consensus.cc:2454] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
W20260806 04:08:24.727324 9471 log.cc:924] Time spent T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260806 04:08:24.735458 9414 consensus_queue.cc:260] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.743769 9316 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.744397 9414 raft_consensus.cc:3037] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:24.746452 9245 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:24.750609 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.728981 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:24.752779 4992 leader_election.cc:334] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857): Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 2.017s (SENT)
I20260806 04:08:24.737360 4924 rpcz_store.cc:276] 0806 04:08:22.556122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.556518 (+ 396us) service_pool.cc:227] Handling call
0806 04:08:22.557061 (+ 543us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:22.597767 (+ 40706us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.597803 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:22.605148 (+ 7345us) log.cc:842] Serialized 26 byte log entry
0806 04:08:23.047482 (+442334us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:23.111864 (+ 64382us) log.cc:842] Serialized 14 byte log entry
0806 04:08:23.121501 (+ 9637us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.736577 (+1615076us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b5400354dd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:24.736625 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:24.736675 (+ 50us) raft_consensus.cc:1671] finished
0806 04:08:24.736775 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.736921 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11536,"prepare.queue_time_us":63660,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":58,"raft.queue_time_us":10899,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":9104,"spinlock_wait_cycles":92615424,"thread_start_us":338940,"threads_started":1,"wal-append.queue_time_us":506097,"wal-append.run_cpu_time_us":2064,"wal-append.run_wall_time_us":1003431}
I20260806 04:08:24.757282 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:24.759351 4992 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:34040 dropped due to backpressure. The service queue is full; it has 50 items.
I20260806 04:08:24.761206 4768 leader_election.cc:398] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 5.
I20260806 04:08:24.764103 4768 leader_election.cc:398] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:24.764385 4768 leader_election.cc:401] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:08:24.759876 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.766337 8975 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.796510 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.797282 5038 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:24.767227 4992 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:34040 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1801)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 358)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 362)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1794)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 364)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1795)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1777)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1775)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1789)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 365)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 366)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1782)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 363)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 359)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 341)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2164)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2169)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2171)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2173)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2174)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2175)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2176)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2177)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2179)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2181)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2182)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1788)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1793)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1797)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 374)
I20260806 04:08:24.809021 4768 leader_election.cc:398] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:24.809525 4768 leader_election.cc:401] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:24.812009 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 498) took 1831 ms. Trace:
I20260806 04:08:24.814342 4809 rpcz_store.cc:276] 0806 04:08:22.980562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.983838 (+ 3276us) service_pool.cc:227] Handling call
0806 04:08:24.811278 (+1827440us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b5800001898. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de8b02b0 00007f23de89b7f9 00007f23de89b2a2 00007f23d7f81684
0806 04:08:24.811487 (+ 209us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:24.811693 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:24.811717 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:24.811758 (+ 41us) raft_consensus.cc:1631] Marking committed up to 10
0806 04:08:24.811794 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.811847 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.811949 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11134336}
I20260806 04:08:24.817811 5812 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:24.821028 9480 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:24.816313 9484 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 5. Current term is 6.
I20260806 04:08:24.804430 9481 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:24.831861 4768 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:24.832933 5030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 344) took 1774 ms (client timeout 1565 ms). Trace:
W20260806 04:08:24.833318 5030 rpcz_store.cc:269] 0806 04:08:23.058390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.107467 (+ 49077us) service_pool.cc:227] Handling call
0806 04:08:24.832731 (+1725264us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400017554. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:24.832864 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2837273344}
I20260806 04:08:24.832031 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2166) took 1600 ms. Trace:
I20260806 04:08:24.834928 5034 rpcz_store.cc:276] 0806 04:08:23.232118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.974538 (+742420us) service_pool.cc:227] Handling call
0806 04:08:24.831830 (+857292us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000bd954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:24.831972 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340481920}
I20260806 04:08:24.803344 9245 consensus_queue.cc:1046] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
W20260806 04:08:24.839232 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 110546us
I20260806 04:08:24.754807 9275 consensus_queue.cc:237] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.841955 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.831142 9195 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:24.843043 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.846074 5034 raft_consensus.cc:2362] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5. Re-sending same reply.
I20260806 04:08:24.843815 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.847016 5030 raft_consensus.cc:3037] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:24.838701 9484 raft_consensus.cc:2781] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:24.836580 9481 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.850174 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:24.851032 9252 raft_consensus.cc:3032] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:24.851517 9252 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.852423 9252 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.853832 9484 raft_consensus.cc:686] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [term 5 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:24.855588 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 333) took 2157 ms. Trace:
I20260806 04:08:24.855962 5035 rpcz_store.cc:276] 0806 04:08:22.697847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.163342 (+1465495us) service_pool.cc:227] Handling call
0806 04:08:24.651066 (+487724us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:24.662736 (+ 11670us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.855352 (+192616us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b540008e650. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:24.855524 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1117894016}
I20260806 04:08:24.857949 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.853801 9252 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:24.861805 4930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1624) took 2398 ms. Trace:
I20260806 04:08:24.862962 9484 consensus_queue.cc:237] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:24.866134 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 953632us
I20260806 04:08:24.866328 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 515) took 1284 ms. Trace:
I20260806 04:08:24.866870 4882 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.867834 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.858455 5034 raft_consensus.cc:2362] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5. Re-sending same reply.
I20260806 04:08:24.869288 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:24.862192 4930 rpcz_store.cc:276] 0806 04:08:22.462979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.463152 (+ 173us) service_pool.cc:227] Handling call
0806 04:08:22.471617 (+ 8465us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:22.472002 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.472030 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:23.590813 (+1118783us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x7b2c0000f228. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d833de11 00007f23d834d442 00007f23d9995c11 00007f23d995a58a 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772
0806 04:08:23.591269 (+ 456us) log.cc:842] Serialized 26 byte log entry
0806 04:08:23.789100 (+197831us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:23.826079 (+ 36979us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.861263 (+1035184us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b180031ee80. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23d99523d0 00007f23d9964576 00007f23d99750d7 00007f23d9975058
0806 04:08:24.861452 (+ 189us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.861528 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:24.861576 (+ 48us) raft_consensus.cc:1671] finished
0806 04:08:24.861588 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.861741 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":660,"prepare.queue_time_us":37045,"prepare.run_cpu_time_us":24591,"prepare.run_wall_time_us":52662,"raft.queue_time_us":6949,"raft.run_cpu_time_us":928,"raft.run_wall_time_us":75831,"spinlock_wait_cycles":30650496,"thread_start_us":146135,"threads_started":1,"wal-append.queue_time_us":396783}
I20260806 04:08:24.866803 4926 rpcz_store.cc:276] 0806 04:08:23.581800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.698064 (+116264us) service_pool.cc:227] Handling call
0806 04:08:23.698682 (+ 618us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:23.699038 (+ 356us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:08:23.741347 (+ 42309us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.866000 (+1124653us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:24.866050 (+ 50us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:24.866095 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.866157 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.866266 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":975038,"prepare.queue_time_us":27835,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"spinlock_wait_cycles":2439552,"thread_start_us":149141,"threads_started":1,"wal-append.queue_time_us":212923}
I20260806 04:08:24.874238 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.884588 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:24.849639 9481 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:24.888175 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.874109 9417 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:24.894605 9417 raft_consensus.cc:686] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 9 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:24.895874 9417 consensus_queue.cc:237] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.901013 5853 raft_consensus.cc:2396] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:24.901942 9243 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:24.903075 9146 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:24.903666 9146 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:24.902578 9243 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.912091 5030 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:24.922214 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.930583 5035 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:24.932166 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.932969 5035 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:24.933981 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.936125 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.897243 9028 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:24.920548 9146 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:24.904582 9291 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:24.945165 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.945297 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.945997 5034 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:24.896996 8936 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20260806 04:08:24.946297 5030 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 9.
I20260806 04:08:24.951198 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.952994 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.954802 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:24.955520 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 119859us
I20260806 04:08:24.955662 5030 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:24.955734 5034 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 8: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:24.961978 5035 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:24.964622 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:24.965561 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:24.966089 4768 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.967166 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:24.967327 4768 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.968418 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:24.968389 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.969907 5035 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:24.970170 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:24.964947 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2146) took 3036 ms. Trace:
I20260806 04:08:24.971010 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.971700 5035 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:24.972683 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.973121 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:24.973286 5035 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:24.974234 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.974727 5035 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:24.975596 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.980851 9498 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:24.981528 9498 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:24.982887 9484 raft_consensus.cc:2781] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:24.983661 9484 raft_consensus.cc:686] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:24.984187 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.984365 8975 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:24.984877 8975 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.984561 9484 consensus_queue.cc:237] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:24.982800 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:24.986757 4768 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:24.987447 4925 raft_consensus.cc:1226] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 04:08:24.971158 5028 rpcz_store.cc:276] 0806 04:08:21.927634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.940028 (+ 12394us) service_pool.cc:227] Handling call
0806 04:08:21.956691 (+ 16663us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:08:22.025679 (+ 68988us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.025727 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 04:08:22.087392 (+ 61665us) log.cc:842] Serialized 78 byte log entry
0806 04:08:24.844339 (+2756947us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b180044fe60. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a14345 00007f23df84ba39 00007f23d6a318bb 00007f23d6a2fcbf 00007f23d6a31e22 00007f23d95db5bb 00007f23d95e1063 00007f23d95e1411 00007f23d992e5cf
0806 04:08:24.878429 (+ 34090us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:24.956349 (+ 77920us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.956604 (+ 255us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:24.956686 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:24.956726 (+ 40us) raft_consensus.cc:1671] finished
0806 04:08:24.963202 (+ 6476us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:24.963550 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68433,"raft.queue_time_us":130805,"raft.run_cpu_time_us":678,"raft.run_wall_time_us":181959,"spinlock_wait_cycles":6375936,"thread_start_us":110559,"threads_started":2,"wal-append.queue_time_us":2618124}
I20260806 04:08:24.996373 4768 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.994508 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:24.996503 5035 raft_consensus.cc:2454] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:24.995409 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:24.985328 8975 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:24.999661 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:24.947635 9291 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.002400 5035 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:24.945675 9028 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.003489 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.005343 4768 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:24.957633 9207 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:24.971122 9153 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:24.994889 9245 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.015491 9291 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:25.017120 9426 log.cc:924] Time spent T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 1.081s user 0.002s sys 0.011s
I20260806 04:08:25.007315 9316 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:25.018234 9316 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:25.018668 9316 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:25.007035 9234 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.019512 5029 raft_consensus.cc:1261] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:08:25.019385 9234 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.020114 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2161) took 3075 ms. Trace:
I20260806 04:08:25.020450 5029 rpcz_store.cc:276] 0806 04:08:21.944295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.018844 (+3074549us) service_pool.cc:227] Handling call
0806 04:08:25.019160 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.019863 (+ 703us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.020060 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.021497 9234 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:24.994081 9480 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:25.012246 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.006775 9251 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.007496 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.026108 4919 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:24.997180 9348 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:25.056525 4931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1627) took 2537 ms (client timeout 3011 ms). Trace:
I20260806 04:08:25.058972 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.051542 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.067059 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.081169 5860 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 9.
I20260806 04:08:25.081489 5035 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:25.041378 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.082365 4770 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.082926 5856 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:25.050108 8997 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.078744 9359 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:08:25.022562 9490 log.cc:924] Time spent T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
W20260806 04:08:25.031222 9488 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260806 04:08:25.028302 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.092029 4879 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:25.053892 9204 consensus_queue.cc:260] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.029942 9146 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.024098 9028 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.105556 9028 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:25.106179 9028 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.107519 9306 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:25.084623 9480 raft_consensus.cc:686] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:25.108042 9306 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:25.067365 9245 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.108562 9306 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:25.084021 9359 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.108991 9480 consensus_queue.cc:237] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.109831 9359 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.106582 9028 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:25.113677 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.114513 4924 raft_consensus.cc:2354] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:25.021438 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.117012 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.117594 5029 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:25.026041 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.122553 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 505) took 1579 ms. Trace:
I20260806 04:08:25.122903 4804 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 04:08:25.123070 4808 rpcz_store.cc:276] 0806 04:08:23.543654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.564131 (+ 20477us) service_pool.cc:227] Handling call
0806 04:08:25.122374 (+1558243us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002710d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:25.122495 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282747776}
I20260806 04:08:25.035820 5854 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:25.125654 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2154) took 3187 ms. Trace:
I20260806 04:08:25.118356 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.126766 9322 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 04:08:25.127359 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2158) took 3184 ms. Trace:
W20260806 04:08:25.128705 5026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 355) took 1872 ms (client timeout 2217 ms). Trace:
W20260806 04:08:25.129277 5026 rpcz_store.cc:269] 0806 04:08:23.256162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.675532 (+419370us) service_pool.cc:227] Handling call
0806 04:08:25.128515 (+1452983us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b5400321ed4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:25.128636 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1842563328}
I20260806 04:08:25.127857 5037 rpcz_store.cc:276] 0806 04:08:21.943348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.487179 (+543831us) service_pool.cc:227] Handling call
0806 04:08:22.506623 (+ 19444us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:22.520452 (+ 13829us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:08:22.520728 (+ 276us) log.cc:842] Serialized 14 byte log entry
0806 04:08:23.965805 (+1445077us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b18004cde00. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23dd32f167 00007f23d9964807 00007f23d9973f0f 00007f23d9973e88
0806 04:08:23.966050 (+ 245us) log.cc:842] Serialized 14 byte log entry
0806 04:08:24.065727 (+ 99677us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:24.065779 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:24.065828 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.127081 (+1061253us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b540008d254. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:25.127143 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:25.127291 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":248349,"prepare.queue_time_us":1080882,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":12041,"raft.queue_time_us":136929,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":19916,"spinlock_wait_cycles":36859648,"thread_start_us":404778,"threads_started":1,"wal-append.queue_time_us":530427,"wal-append.run_cpu_time_us":2173,"wal-append.run_wall_time_us":1908567}
I20260806 04:08:25.126037 5036 rpcz_store.cc:276] 0806 04:08:21.937951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.471470 (+533519us) service_pool.cc:227] Handling call
0806 04:08:22.538429 (+ 66959us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:22.538749 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:22.538774 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:22.539539 (+ 765us) log.cc:842] Serialized 26 byte log entry
0806 04:08:22.649598 (+110059us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:23.086305 (+436707us) log.cc:842] Serialized 14 byte log entry
0806 04:08:23.093362 (+ 7057us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:23.093438 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:23.238734 (+145296us) raft_consensus.cc:1671] finished
0806 04:08:23.238755 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:25.125495 (+1886740us) spinlock_profiling.cc:243] Waited 66.6 ms on lock 0x7b54002d1ed0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d9959a09 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:25.125594 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1173210,"prepare.queue_time_us":272172,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":16138,"raft.queue_time_us":13155,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":10089,"spinlock_wait_cycles":153225088,"thread_start_us":370014,"threads_started":2,"wal-append.queue_time_us":284201,"wal-append.run_cpu_time_us":4903,"wal-append.run_wall_time_us":1534197}
I20260806 04:08:25.131321 5037 raft_consensus.cc:1261] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:08:25.127619 9322 consensus_queue.cc:1225] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer d7c76b6dcd8945cd818b8107caf4a174 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260806 04:08:25.132251 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2162) took 1918 ms. Trace:
I20260806 04:08:25.132467 5036 raft_consensus.cc:1226] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:25.144320 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2163) took 1929 ms. Trace:
I20260806 04:08:25.023154 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59636 (request call id 718) took 3257 ms. Trace:
I20260806 04:08:25.083994 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.039083 9498 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.083555 9251 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.147033 5856 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:25.053546 9243 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.008047 9169 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:25.148231 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.148639 9169 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:25.149243 9169 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:25.115445 9340 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
W20260806 04:08:25.132736 9448 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.178s user 0.003s sys 0.007s
I20260806 04:08:25.132674 5037 rpcz_store.cc:276] 0806 04:08:23.214057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.130414 (+1916357us) service_pool.cc:227] Handling call
0806 04:08:25.130915 (+ 501us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.131976 (+ 1061us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.132187 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.126372 5029 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:25.050880 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:25.134765 4928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1613) took 2859 ms (client timeout 2017 ms). Trace:
W20260806 04:08:25.139317 9431 log.cc:924] Time spent T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 1.209s user 0.004s sys 0.006s
I20260806 04:08:25.141270 4927 raft_consensus.cc:1226] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:25.142016 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:25.143189 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1639) took 2045 ms. Trace:
I20260806 04:08:25.145588 9450 raft_consensus.cc:3032] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:25.147727 5859 rpcz_store.cc:276] 0806 04:08:21.765921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:21.767983 (+ 2062us) service_pool.cc:227] Handling call
0806 04:08:21.794136 (+ 26153us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:23.099126 (+1304990us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.022918 (+1923792us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540051dbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:25.023092 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2967129472}
I20260806 04:08:25.149363 5038 raft_consensus.cc:2379] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:25.175614 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2169) took 1906 ms. Trace:
W20260806 04:08:25.084808 4931 rpcz_store.cc:269] 0806 04:08:22.519035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.519411 (+ 376us) service_pool.cc:227] Handling call
0806 04:08:25.056262 (+2536851us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b540006e854. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:25.056443 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31739776}
I20260806 04:08:25.137889 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:25.158484 4928 rpcz_store.cc:269] 0806 04:08:22.275579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.275711 (+ 132us) service_pool.cc:227] Handling call
0806 04:08:25.134557 (+2858846us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54004141d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:25.134696 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":633545600}
I20260806 04:08:25.131363 5026 raft_consensus.cc:1261] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:25.153475 5036 rpcz_store.cc:276] 0806 04:08:23.214476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.131939 (+1917463us) service_pool.cc:227] Handling call
0806 04:08:25.132246 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.144005 (+ 11759us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.144249 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.048583 9130 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.085881 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.028769 9436 raft_consensus.cc:3032] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 04:08:25.208422 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 598231us
I20260806 04:08:25.210549 5035 raft_consensus.cc:1226] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:25.176082 5038 rpcz_store.cc:276] 0806 04:08:23.268798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.174505 (+1905707us) service_pool.cc:227] Handling call
0806 04:08:25.174915 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.175168 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.175207 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:25.175263 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:25.175312 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.175380 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:25.175542 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.211428 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2173) took 1167 ms. Trace:
I20260806 04:08:25.177520 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 341) took 2163 ms. Trace:
I20260806 04:08:25.211783 5035 rpcz_store.cc:276] 0806 04:08:24.044011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.210066 (+1166055us) service_pool.cc:227] Handling call
0806 04:08:25.210376 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.211202 (+ 826us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.211375 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:25.077417 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 69986.1us
I20260806 04:08:25.008350 9479 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:25.165033 4920 rpcz_store.cc:276] 0806 04:08:23.098040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.151930 (+ 53890us) service_pool.cc:227] Handling call
0806 04:08:23.152244 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:23.352891 (+200647us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.141874 (+1788983us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54004132d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:25.143098 (+ 1224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459834752}
I20260806 04:08:25.187633 5036 raft_consensus.cc:1226] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:25.212323 5026 rpcz_store.cc:276] 0806 04:08:23.013885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.130361 (+2116476us) service_pool.cc:227] Handling call
0806 04:08:25.131046 (+ 685us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.177191 (+ 46145us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.177454 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.213318 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.172920 5863 raft_consensus.cc:2396] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:25.212297 4808 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:25.214309 5026 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260806 04:08:25.218859 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 97733.1us
I20260806 04:08:25.223245 4768 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.225616 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2171) took 1182 ms. Trace:
I20260806 04:08:25.225800 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.226197 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.214411 5035 raft_consensus.cc:1226] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:25.227144 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2175) took 1179 ms. Trace:
W20260806 04:08:25.228965 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 362152us
W20260806 04:08:25.229277 5863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 719) took 1708 ms (client timeout 1593 ms). Trace:
I20260806 04:08:25.226081 5036 rpcz_store.cc:276] 0806 04:08:24.043495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.187125 (+1143630us) service_pool.cc:227] Handling call
0806 04:08:25.187463 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.221476 (+ 34013us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.225539 (+ 4063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.227551 5035 rpcz_store.cc:276] 0806 04:08:24.048029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.212555 (+1164526us) service_pool.cc:227] Handling call
0806 04:08:25.212839 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.226900 (+ 14061us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.227088 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.230918 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:25.232064 5037 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:25.229866 5863 rpcz_store.cc:269] 0806 04:08:23.521030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.521257 (+ 227us) service_pool.cc:227] Handling call
0806 04:08:25.141805 (+1620548us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5800013e98. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d69fa851 00007f23d69fa702 00007f23df8eab09 00007f23de826730 00007f23de8ec79b 00007f23de8ec821 00007f23de89c1b0 00007f23de89ba94 00007f23d7f83c04 00007f23d7f83b92
0806 04:08:25.229203 (+ 87398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3255168}
I20260806 04:08:25.224933 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2176) took 1176 ms. Trace:
I20260806 04:08:25.235718 5026 rpcz_store.cc:276] 0806 04:08:24.048413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.213522 (+1165109us) service_pool.cc:227] Handling call
0806 04:08:25.213875 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.214859 (+ 984us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.224867 (+ 10008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.237140 5026 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:25.010382 9207 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.244980 5035 raft_consensus.cc:1261] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:25.173507 9450 raft_consensus.cc:729] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:25.246137 4768 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.246279 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2177) took 1197 ms. Trace:
I20260806 04:08:25.247634 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:25.268841 4814 raft_consensus.cc:1226] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:25.248528 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:25.257436 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2179) took 1207 ms. Trace:
I20260806 04:08:25.245332 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2181) took 1179 ms. Trace:
I20260806 04:08:25.270941 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2180) took 1212 ms. Trace:
I20260806 04:08:25.271548 5032 rpcz_store.cc:276] 0806 04:08:24.058910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.235510 (+1176600us) service_pool.cc:227] Handling call
0806 04:08:25.270330 (+ 34820us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.270562 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.270601 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:25.270673 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:25.270732 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.270789 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:25.270888 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.270203 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:25.224921 9258 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 10 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.227538 9252 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.273887 9146 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.278764 4920 raft_consensus.cc:1261] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260806 04:08:25.277529 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2164) took 2062 ms. Trace:
I20260806 04:08:25.207798 9436 raft_consensus.cc:729] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:25.276175 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:25.256300 8797 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:25.267549 5036 rpcz_store.cc:276] 0806 04:08:24.048742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.245529 (+1196787us) service_pool.cc:227] Handling call
0806 04:08:25.245821 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.245991 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.246013 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:25.246049 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:25.246082 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.246133 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:25.246228 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.266430 9478 raft_consensus.cc:2781] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:25.234503 9340 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:25.295243 4876 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:25.306319 9343 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.306205 8975 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.307124 9343 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.281201 9157 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260806 04:08:25.311117 4805 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:25.311267 4816 raft_consensus.cc:1261] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 04:08:25.228643 8921 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260806 04:08:25.186866 3041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:08:25.278127 9002 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:25.280885 5037 rpcz_store.cc:276] 0806 04:08:23.214848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.156513 (+1941665us) service_pool.cc:227] Handling call
0806 04:08:25.231631 (+ 75118us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.255886 (+ 24255us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.277449 (+ 21563us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.281883 5034 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:25.245363 9410 consensus_queue.cc:1035] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260806 04:08:25.262938 5029 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:25.259161 9479 raft_consensus.cc:686] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:25.258872 9322 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.270812 5035 rpcz_store.cc:276] 0806 04:08:24.050251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.244195 (+1193944us) service_pool.cc:227] Handling call
0806 04:08:25.244551 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.257095 (+ 12544us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.257379 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.289249 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.267546 9444 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.294781 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.295758 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.271173 9204 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:25.282110 9402 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.540s user 0.015s sys 0.020s
I20260806 04:08:25.299532 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.300863 9443 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.270923 5026 rpcz_store.cc:276] 0806 04:08:24.065969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.236479 (+1170510us) service_pool.cc:227] Handling call
0806 04:08:25.236789 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.237460 (+ 671us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.245270 (+ 7810us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.370738 9443 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.371572 5026 raft_consensus.cc:1261] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:08:25.371706 9343 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.320474 5034 raft_consensus.cc:1261] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260806 04:08:25.372519 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.372921 9443 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.372967 5034 pending_rounds.cc:76] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Aborting all ops after (but not including) 5
I20260806 04:08:25.373764 5034 pending_rounds.cc:97] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 04:08:25.373703 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.374200 5034 raft_consensus.cc:2866] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:25.371582 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.375540 4920 raft_consensus.cc:2379] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 8: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:25.326666 5029 raft_consensus.cc:1226] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:25.288707 8797 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.334004 5036 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.291378 9345 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:08:25.352842 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.388139 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.389020 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.273954 9252 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.358141 5035 raft_consensus.cc:1226] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260806 04:08:25.356177 9402 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.641s user 0.015s sys 0.021s
I20260806 04:08:25.356295 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.357506 9322 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.361626 9479 consensus_queue.cc:237] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.408180 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:25.365502 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 70368.5us
I20260806 04:08:25.383594 8797 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:25.366369 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.366616 9357 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:25.317149 9245 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:25.367138 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.317750 9478 raft_consensus.cc:686] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:25.367717 8180 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.367916 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.368039 8641 raft_consensus.cc:2726] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:08:25.357864 9444 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.368978 4925 raft_consensus.cc:1226] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 2. Current term is 3. Ops: []
I20260806 04:08:25.392249 9157 consensus_queue.cc:1046] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260806 04:08:25.369582 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.369891 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.375010 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2182) took 1299 ms. Trace:
I20260806 04:08:25.428997 4919 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:08:25.428938 5034 rpcz_store.cc:276] 0806 04:08:24.075337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.319450 (+1244113us) service_pool.cc:227] Handling call
0806 04:08:25.319779 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.374757 (+ 54978us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.374954 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:25.430184 4924 raft_consensus.cc:2396] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 04:08:25.293440 9360 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:25.313212 9340 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:25.386811 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.404850 9402 raft_consensus.cc:686] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.399395 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.321278 8863 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:25.402868 5859 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:08:25.332996 9002 raft_consensus.cc:686] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:25.358464 4931 raft_consensus.cc:2396] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 04:08:25.404160 5037 raft_consensus.cc:1226] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:25.451220 5037 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:08:25.351668 9303 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:25.273610 9251 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.455480 9359 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.408365 4882 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:25.365614 9169 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.436363 9340 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.458714 9169 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.459414 4804 raft_consensus.cc:1226] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:25.462235 9359 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.463263 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.463811 4931 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:25.464466 9207 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:25.464818 4931 raft_consensus.cc:1226] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:25.465590 9157 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:08:25.466408 9429 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:25.410077 4926 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:25.420487 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:25.475438 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 67149.2us
I20260806 04:08:25.475952 5035 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.433202 4927 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:08:25.434485 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.476701 4930 raft_consensus.cc:3037] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.478094 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2390) took 2389 ms. Trace:
I20260806 04:08:25.479043 4816 rpcz_store.cc:276] 0806 04:08:23.089224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:23.089570 (+ 346us) service_pool.cc:227] Handling call
0806 04:08:23.090008 (+ 438us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.311829 (+2221821us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.477860 (+166031us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b5400270bd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:25.478034 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5107276032}
I20260806 04:08:25.480731 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.436246 5029 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:08:25.390552 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:25.487742 4814 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:25.437114 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.415102 8975 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.443001 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.442307 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.491334 4928 raft_consensus.cc:2379] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 10: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:25.446797 4924 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260806 04:08:25.481459 4816 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:25.425279 9195 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.452095 5863 raft_consensus.cc:1261] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:08:25.414983 9498 consensus_queue.cc:1046] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260806 04:08:25.432430 8180 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.422053 9191 consensus_queue.cc:1035] T 7cb317a04be74515913f7aee753c193b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
W20260806 04:08:25.454236 9506 log.cc:924] Time spent T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260806 04:08:25.453958 9002 consensus_queue.cc:237] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.425726 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:25.383168 9316 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.493492 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:25.494374 4996 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:25.556690 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 62407.8us
I20260806 04:08:25.427249 9357 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.498381 5035 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:25.563265 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.506254 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.564283 5029 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.507217 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.505173 9510 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
W20260806 04:08:25.490257 9481 consensus_meta.cc:287] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.569s user 0.015s sys 0.000s
I20260806 04:08:25.534206 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.573976 4805 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:25.478999 9251 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:25.523725 5030 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:25.548192 4814 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:25.576272 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.439131 9360 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:25.552989 4651 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 287001us
I20260806 04:08:25.449167 5036 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:25.592653 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.593689 5036 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:08:25.594715 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.595122 5026 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:25.595986 4818 raft_consensus.cc:2379] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 10: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:25.558485 4816 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:08:25.567130 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 60016.8us
W20260806 04:08:25.567251 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 141618us
I20260806 04:08:25.555538 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.529520 9196 raft_consensus.cc:3032] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:25.573804 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.529368 8180 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.526439 9359 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:25.579918 4882 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.534936 9195 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.560206 9514 consensus_queue.cc:1046] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
W20260806 04:08:25.581864 9513 log.cc:924] Time spent T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260806 04:08:25.570302 9501 consensus_queue.cc:1035] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260806 04:08:25.535418 9306 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.575022 9402 consensus_queue.cc:237] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.587370 8797 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.536550 8975 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 04:08:25.683847 9195 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.685943 4925 raft_consensus.cc:2379] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 9: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:25.588703 9252 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.590505 9397 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.531809 9444 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.596489 5036 raft_consensus.cc:1261] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 04:08:25.606276 5859 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:25.606863 4768 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.577540 9169 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:25.607406 9196 raft_consensus.cc:729] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.608500 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.570986 9327 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:25.609100 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.607822 9191 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260806 04:08:25.607422 4876 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:25.718605 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 111236us
I20260806 04:08:25.684656 9498 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:25.719542 9498 raft_consensus.cc:686] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.719657 4876 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:25.720705 9243 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:25.580016 4930 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:25.611033 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.721933 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.615502 9322 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.616909 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.722978 5863 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:25.533079 9450 consensus_queue.cc:260] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.630748 5856 raft_consensus.cc:1226] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:08:25.453186 8863 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.725562 9196 consensus_queue.cc:260] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.668828 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.727022 9196 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:25.727823 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.729001 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.672598 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.730736 4805 raft_consensus.cc:2396] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:08:25.731106 9293 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:25.731930 9243 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:25.673638 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 93809.1us
I20260806 04:08:25.732918 9243 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.733707 4804 raft_consensus.cc:1226] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:25.679019 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.681197 4816 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 10.
I20260806 04:08:25.736747 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.736918 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.738370 5816 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:25.738828 9345 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.708253 9169 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.739944 9169 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:25.739398 9345 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.740845 9252 consensus_queue.cc:1035] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:08:25.742827 9195 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:25.743436 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:25.687620 4919 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:25.745460 9450 raft_consensus.cc:3037] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:25.746215 9357 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.680698 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.747457 4818 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:25.747851 9360 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.709048 9397 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.749006 4882 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:25.749596 9306 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.750084 9397 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.750730 5857 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:08:25.751083 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:25.704674 9156 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.752413 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.752743 9156 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.753907 4814 raft_consensus.cc:1261] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 04:08:25.754311 4814 pending_rounds.cc:76] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 1
I20260806 04:08:25.754544 4814 pending_rounds.cc:97] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 04:08:25.754851 4814 raft_consensus.cc:2866] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:25.712663 9327 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:25.755841 9327 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:25.671602 9359 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:25.712955 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 106275us
I20260806 04:08:25.687041 8180 raft_consensus.cc:3032] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:25.609792 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.720503 9498 consensus_queue.cc:237] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.722033 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.613778 9316 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.592957 9481 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.726122 8863 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.539448 9478 consensus_queue.cc:237] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.752003 9436 consensus_queue.cc:260] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.709878 4995 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:25.821435 9436 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:25.822018 9370 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:25.822093 8995 consensus_queue.cc:1046] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:08:25.822607 9370 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.823146 9370 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:25.715595 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.762691 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:25.763561 4804 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260806 04:08:25.827154 8863 raft_consensus.cc:3032] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:25.827443 4804 pending_rounds.cc:76] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting all ops after (but not including) 5
I20260806 04:08:25.827783 4804 pending_rounds.cc:97] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260806 04:08:25.827714 8863 raft_consensus.cc:729] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.828192 4804 raft_consensus.cc:2866] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:08:25.821655 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 111869us
I20260806 04:08:25.827153 5036 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:25.764704 4818 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:25.830123 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.830804 5036 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 3.
I20260806 04:08:25.831120 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:25.831434 5854 raft_consensus.cc:1261] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:08:25.831598 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.723423 8797 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.832948 5857 raft_consensus.cc:1226] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:25.833930 8921 consensus_queue.cc:1046] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:08:25.742269 9252 consensus_queue.cc:1046] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:08:25.832187 5036 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 9.
I20260806 04:08:25.835795 9252 raft_consensus.cc:3032] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:25.836205 9252 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:25.837556 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:25.837221 9252 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.829926 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.839048 4804 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:25.840469 4804 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260806 04:08:25.841945 9525 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.094s user 0.004s sys 0.000s
W20260806 04:08:25.712663 9028 consensus_meta.cc:287] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.601s user 0.014s sys 0.007s
I20260806 04:08:25.792109 9480 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:25.788292 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:25.844753 9209 consensus_queue.cc:1046] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 04:08:25.810493 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:25.801236 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:25.847515 9209 raft_consensus.cc:3032] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:25.847957 9209 raft_consensus.cc:729] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.847828 8863 consensus_queue.cc:260] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.849323 8863 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.820144 9303 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:08:25.820326 9505 consensus_queue.cc:1046] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:08:25.845182 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2174) took 1800 ms. Trace:
I20260806 04:08:25.850204 9505 raft_consensus.cc:3032] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:25.786271 4996 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.834895 8921 raft_consensus.cc:3032] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:08:25.854415 9360 consensus_queue.cc:1046] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:08:25.847527 5035 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:25.855729 9360 consensus_queue.cc:1035] T 654e437ca202414286305ce8a8c5c4c2 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:08:25.782531 4919 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:25.848902 9209 consensus_queue.cc:260] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.819442 9234 consensus_queue.cc:1035] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:25.859782 9359 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:25.785199 5029 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:25.850128 5038 rpcz_store.cc:276] 0806 04:08:24.044280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.211432 (+1167152us) service_pool.cc:227] Handling call
0806 04:08:25.211818 (+ 386us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:25.225124 (+ 13306us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.225156 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:25.270327 (+ 45171us) log.cc:842] Serialized 52 byte log entry
0806 04:08:25.494566 (+224239us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:25.536830 (+ 42264us) log.cc:842] Serialized 14 byte log entry
0806 04:08:25.537174 (+ 344us) log.cc:842] Serialized 14 byte log entry
0806 04:08:25.537328 (+ 154us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.537396 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:25.726301 (+188905us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b1800417d00. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32764 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:25.741247 (+ 14946us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b1800417d00. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32638 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:25.841183 (+ 99936us) spinlock_profiling.cc:243] Waited 91.1 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:25.844961 (+ 3778us) raft_consensus.cc:1671] finished
0806 04:08:25.844983 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:25.845124 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9130,"prepare.queue_time_us":101418,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":10510,"raft.queue_time_us":32372,"raft.run_cpu_time_us":403,"raft.run_wall_time_us":26065,"spinlock_wait_cycles":241719296,"thread_start_us":197269,"threads_started":1,"wal-append.queue_time_us":421091}
I20260806 04:08:25.860283 9234 consensus_queue.cc:1225] T 8570417a88db412294c21c9108061c30 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer 99690415a7b3441fbdb5e943667556a4 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:08:25.845630 9357 consensus_queue.cc:1035] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
I20260806 04:08:25.826809 9417 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:25.852950 8812 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:08:25.844391 8797 consensus_queue.cc:1046] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260806 04:08:25.854821 8921 raft_consensus.cc:729] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:25.838482 8532 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:25.857631 9443 raft_consensus.cc:3032] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:25.845329 9480 raft_consensus.cc:686] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:25.850709 9505 raft_consensus.cc:729] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.775085 8180 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.842592 4804 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:08:25.865557 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.844535 9028 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:25.848089 9524 log.cc:924] Time spent T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260806 04:08:25.891666 4768 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:25.812924 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 57207.8us
I20260806 04:08:25.903455 4805 raft_consensus.cc:1226] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:25.869781 5035 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 10.
I20260806 04:08:25.872033 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:25.858525 9156 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:25.898272 4926 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:25.881570 8532 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.920904 4930 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 3. Current term is 5. Ops: []
I20260806 04:08:25.720564 9340 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.905097 4804 raft_consensus.cc:1261] T 5854d52dcd124250a917b57b14dd2afc P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:08:25.906723 5029 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:25.923430 5029 raft_consensus.cc:1261] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:08:25.906991 5035 raft_consensus.cc:1261] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:08:25.907419 4768 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.872063 9345 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.887856 9443 raft_consensus.cc:729] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:25.889444 9245 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:25.913056 8921 consensus_queue.cc:260] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.753800 9510 consensus_queue.cc:1035] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.134s
I20260806 04:08:25.799572 9258 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:25.900417 9306 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:25.925931 8532 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:25.963260 9306 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.963522 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:25.963760 9306 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:25.964195 5867 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:25.872428 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 86280.1us
I20260806 04:08:25.926930 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:25.971122 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 44539.3us
I20260806 04:08:25.928056 5028 raft_consensus.cc:2354] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:08:25.972833 9303 consensus_queue.cc:1035] T 4219680d51af4d0587436328294ba316 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:08:25.974357 9303 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:08:25.978977 4928 raft_consensus.cc:1261] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260806 04:08:25.980753 4674 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46902 (request call id 284) took 30706 ms (client timeout 15000 ms). Trace:
I20260806 04:08:25.911710 9359 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:25.981457 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.928795 4804 raft_consensus.cc:1226] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:25.942925 4930 raft_consensus.cc:1226] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 04:08:25.970341 4996 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:25.932793 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:25.993046 5032 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.004s sys 0.033s
W20260806 04:08:25.993515 5032 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s user 0.004s sys 0.033s
I20260806 04:08:25.954049 4808 raft_consensus.cc:1226] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:25.941260 9245 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:25.944711 9480 consensus_queue.cc:237] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:25.996922 4920 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.018s sys 0.013s
I20260806 04:08:25.957639 5856 raft_consensus.cc:1261] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260806 04:08:25.958271 9234 raft_consensus.cc:3032] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:25.932631 8180 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.938604 9209 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:25.959889 9028 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:25.937379 9343 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:25.926795 9195 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260806 04:08:25.962044 9443 consensus_queue.cc:260] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.889937 9417 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:25.980890 4925 raft_consensus.cc:1226] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:25.980614 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.981536 9359 raft_consensus.cc:686] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:25.935638 4926 raft_consensus.cc:1261] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 04:08:25.997310 9245 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.902616 9252 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:25.955554 9370 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:25.997915 4920 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.517s user 0.018s sys 0.013s
W20260806 04:08:25.982286 4674 rpcz_store.cc:269] 0806 04:07:55.274301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:07:55.275251 (+ 950us) service_pool.cc:227] Handling call
0806 04:07:55.528612 (+253361us) spinlock_profiling.cc:243] Waited 68.9 ms on lock 0x7b54001c1954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23df8b4d6c 00007f23df9aec3e 00007f23ddcc0fb4 00007f23ddcc0f42 00007f23ddcc0e71 00007f23ddcc0dec 00007f23ddcc00a2 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:25.980621 (+30452009us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:08:19.685900 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:08:19.831131 (+145231us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:08:19.831185 (+ 54us) write_op.cc:435] Acquired schema lock
0806 04:08:19.831222 (+ 37us) tablet.cc:660] Decoding operations
0806 04:08:19.832228 (+ 1006us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:08:19.832336 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0806 04:08:19.832371 (+ 35us) tablet.cc:683] Acquiring locks for 8 operations
0806 04:08:19.833198 (+ 827us) tablet.cc:699] Row locks acquired
0806 04:08:19.833236 (+ 38us) write_op.cc:260] PREPARE: finished
0806 04:08:19.833415 (+ 179us) write_op.cc:270] Start()
0806 04:08:19.833570 (+ 155us) write_op.cc:276] Timestamp: P: 1785989299833350 usec, L: 0
0806 04:08:19.833606 (+ 36us) op_driver.cc:331] REPLICATION: starting
0806 04:08:19.834322 (+ 716us) log.cc:842] Serialized 3281 byte log entry
0806 04:08:21.751210 (+1916888us) op_driver.cc:444] REPLICATION: finished
0806 04:08:24.464887 (+2713677us) spinlock_profiling.cc:243] Waited 73.2 ms on lock 0x7b18001fd920. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d6a32638 00007f23d6a35b11 00007f23d6a35ac9 00007f23d6a35a59 00007f23d6a35a21 00007f23d6a34d1d 00007f23dfc96204 00007f23dfc96039 00007f23d6a14dc6
0806 04:08:24.482226 (+ 17339us) write_op.cc:301] APPLY: starting
0806 04:08:24.497031 (+ 14805us) tablet.cc:1365] starting BulkCheckPresence
0806 04:08:24.497374 (+ 343us) tablet.cc:1368] finished BulkCheckPresence
0806 04:08:24.497408 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:08:24.548614 (+ 51206us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:08:24.549081 (+ 467us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0806 04:08:24.549154 (+ 73us) write_op.cc:312] APPLY: finished
0806 04:08:24.549632 (+ 478us) log.cc:842] Serialized 65 byte log entry
0806 04:08:24.622681 (+ 73049us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:08:24.627313 (+ 4632us) write_op.cc:454] Released schema lock
0806 04:08:24.638735 (+ 11422us) write_op.cc:341] FINISH: Updating metrics
0806 04:08:24.667381 (+ 28646us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:08:24.677229 (+ 9848us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25511,"reported_tablets":27,"spinlock_wait_cycles":158458624,"child_traces":[["op",{"apply.queue_time_us":349911,"apply.run_cpu_time_us":28862,"apply.run_wall_time_us":214056,"num_ops":8,"prepare.queue_time_us":835771,"prepare.run_cpu_time_us":22367,"prepare.run_wall_time_us":377060,"raft.queue_time_us":1075284,"raft.run_cpu_time_us":18055,"raft.run_wall_time_us":261373,"replication_time_us":1880391,"spinlock_wait_cycles":168332672,"thread_start_us":1816109,"threads_started":5,"wal-append.queue_time_us":365327,"wal-append.run_cpu_time_us":49340,"wal-append.run_wall_time_us":3568785}]]}
I20260806 04:08:25.984450 9327 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:25.915154 9204 consensus_queue.cc:1035] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260806 04:08:25.981473 9275 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.083936 5867 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:26.004576 5856 pending_rounds.cc:76] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Aborting all ops after (but not including) 5
I20260806 04:08:26.054283 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:26.038614 9303 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:26.004724 9130 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:26.004880 9234 raft_consensus.cc:729] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:26.056185 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 086f8af9433f44aa8aa4fccbfa90eb02 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:26.054467 9153 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.073630 4668 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:26.054673 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1782) took 1421 ms. Trace:
W20260806 04:08:26.131630 4651 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 88722.2us
I20260806 04:08:26.132016 5028 rpcz_store.cc:276] 0806 04:08:24.633019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:24.997682 (+364663us) service_pool.cc:227] Handling call
0806 04:08:26.039971 (+1042289us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b54003fc0d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:26.054585 (+ 14614us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1700242816}
I20260806 04:08:26.041791 9343 raft_consensus.cc:686] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 10 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:26.107942 4879 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
W20260806 04:08:26.056772 4923 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.501s user 0.011s sys 0.028s
I20260806 04:08:25.982640 9481 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.132848 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:26.169821 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1625) took 3679 ms. Trace:
I20260806 04:08:26.032739 9433 consensus_queue.cc:1035] T 5854d52dcd124250a917b57b14dd2afc P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260806 04:08:26.096287 9443 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:26.162793 4926 raft_consensus.cc:1261] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:08:26.126825 5856 pending_rounds.cc:97] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 04:08:26.164978 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:25.951359 9505 consensus_queue.cc:260] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.168216 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.080200 9370 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:26.068220 8180 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:26.168913 4668 catalog_manager.cc:5495] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:26.183084 9234 consensus_queue.cc:260] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.162717 8533 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:26.183638 8532 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.184576 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:26.186128 9343 consensus_queue.cc:237] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.170516 4923 rpcz_store.cc:276] 0806 04:08:22.490929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:22.492121 (+ 1192us) service_pool.cc:227] Handling call
0806 04:08:22.507075 (+ 14954us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:22.529586 (+ 22511us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:08:22.925605 (+396019us) log.cc:842] Serialized 14 byte log entry
0806 04:08:23.317814 (+392209us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.121346 (+1803532us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b2c0000f228. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d833de11 00007f23d834d442 00007f23d9995c11 00007f23d995a58a 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772
0806 04:08:25.121704 (+ 358us) log.cc:842] Serialized 26 byte log entry
0806 04:08:25.548299 (+426595us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:26.160908 (+612609us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.161199 (+ 291us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.161347 (+ 148us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.169589 (+ 8242us) raft_consensus.cc:1671] finished
0806 04:08:26.169613 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.169757 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35179,"prepare.queue_time_us":218439,"prepare.run_cpu_time_us":7336,"prepare.run_wall_time_us":41927,"raft.queue_time_us":54180,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":20871,"spinlock_wait_cycles":23796608,"thread_start_us":1679231,"threads_started":4,"wal-append.queue_time_us":1155612,"wal-append.run_cpu_time_us":1693,"wal-append.run_wall_time_us":1079240}
I20260806 04:08:26.073490 9306 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.188405 5034 raft_consensus.cc:1261] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:08:26.079450 9169 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.192413 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:26.166602 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.128669 9130 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:26.210139 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 387752us
I20260806 04:08:26.218511 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:26.159685 9258 raft_consensus.cc:2781] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:26.232529 4927 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 10.
I20260806 04:08:26.236218 9130 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:26.213109 5856 raft_consensus.cc:2866] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:26.126456 9275 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.233024 4668 catalog_manager.cc:5495] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 2, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 2 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:26.074542 9359 consensus_queue.cc:237] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.133651 9303 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.161928 9153 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:26.234802 9251 consensus_meta.cc:287] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.635s user 0.009s sys 0.001s
I20260806 04:08:26.248365 4804 raft_consensus.cc:1226] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
W20260806 04:08:26.236819 9528 log.cc:924] Time spent T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 04:08:26.249739 4668 catalog_manager.cc:5495] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 5, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 5 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:26.254225 4882 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.236718 9258 raft_consensus.cc:686] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:26.262246 4882 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.264228 4668 catalog_manager.cc:5495] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:26.270087 8797 raft_consensus.cc:2726] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:08:26.274423 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 192577us
I20260806 04:08:26.275601 9157 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.276132 9157 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.278995 5866 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 04:08:26.251307 9234 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:26.292039 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 438525us
I20260806 04:08:26.256125 9251 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:26.278453 9275 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.267524 8995 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:08:26.248366 9293 consensus_queue.cc:1035] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260806 04:08:26.265756 9340 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.302263 4668 catalog_manager.cc:5495] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:26.307791 9169 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.261706 9357 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 04:08:26.322491 9157 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.308609 9327 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.361722 4809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2415) took 1028 ms. Trace:
I20260806 04:08:26.314049 4876 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:26.367604 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 53649.1us
I20260806 04:08:26.366631 4809 rpcz_store.cc:276] 0806 04:08:25.333056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.333672 (+ 616us) service_pool.cc:227] Handling call
0806 04:08:25.334021 (+ 349us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:25.356537 (+ 22516us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.356568 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:25.381620 (+ 25052us) log.cc:842] Serialized 52 byte log entry
0806 04:08:25.793280 (+411660us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:25.795144 (+ 1864us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:25.795269 (+ 125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.361484 (+566215us) raft_consensus.cc:1671] finished
0806 04:08:26.361506 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.361654 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22721,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8747,"thread_start_us":371142,"threads_started":1,"wal-append.queue_time_us":638237}
I20260806 04:08:26.300414 9510 consensus_queue.cc:1046] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260806 04:08:26.315019 9417 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.334240 9316 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:26.372787 4668 catalog_manager.cc:5495] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 4, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 4 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:26.376690 4876 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:26.378861 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 391068us
W20260806 04:08:26.370404 9402 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.027s sys 0.028s
I20260806 04:08:26.317596 9433 consensus_queue.cc:1046] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260806 04:08:26.300446 9146 consensus_queue.cc:1046] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260806 04:08:26.393302 4876 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.373199 9357 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:26.394428 9153 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.394820 9544 raft_consensus.cc:2726] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:08:26.394834 9514 consensus_queue.cc:1035] T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 04:08:26.386821 9436 consensus_meta.cc:287] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.565s user 0.006s sys 0.006s
I20260806 04:08:26.397210 4668 catalog_manager.cc:5495] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 3, leader changed from df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 3 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
W20260806 04:08:26.378145 9501 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.006s sys 0.038s
I20260806 04:08:26.387374 4768 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:26.391623 9402 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.027s sys 0.029s
I20260806 04:08:26.425017 9402 consensus_queue.cc:1046] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:08:26.426290 9402 raft_consensus.cc:3032] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 04:08:26.427369 4814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.002s sys 0.020s
W20260806 04:08:26.428117 4814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s user 0.002s sys 0.020s
I20260806 04:08:26.431605 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45408 (request call id 1752) took 1067 ms. Trace:
I20260806 04:08:26.369169 9258 consensus_queue.cc:237] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:26.421386 9501 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.565s user 0.007s sys 0.038s
I20260806 04:08:26.408631 9510 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 04:08:26.436587 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 49290.7us
W20260806 04:08:26.438419 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.000s sys 0.021s
W20260806 04:08:26.444923 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.001s sys 0.021s
I20260806 04:08:26.437546 4668 catalog_manager.cc:5495] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 4, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 4 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
W20260806 04:08:26.445318 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 507974us
I20260806 04:08:26.419103 9306 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.443130 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.407618 9327 raft_consensus.cc:3032] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:26.446419 9570 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:26.443929 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 353266us
W20260806 04:08:26.433485 8863 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.552s user 0.005s sys 0.007s
I20260806 04:08:26.434012 4920 rpcz_store.cc:276] 0806 04:08:25.364577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.444917 (+ 80340us) service_pool.cc:227] Handling call
0806 04:08:25.479970 (+ 35053us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.480232 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:26.066172 (+585940us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.430607 (+364435us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:26.430707 (+ 100us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:26.430790 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.431403 (+ 613us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.431534 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":758193,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":2718,"raft.queue_time_us":90,"thread_start_us":949631,"threads_started":2}
I20260806 04:08:26.452440 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:26.432493 8641 raft_consensus.cc:2781] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:26.452646 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.433975 9303 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:26.453423 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:26.450214 4668 catalog_manager.cc:5495] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 4, leader changed from 99690415a7b3441fbdb5e943667556a4 (127.2.248.67) to 22fe557f852d4415a96fd683f970cc24 (127.2.248.66). New cstate: current_term: 4 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:26.459378 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:26.456564 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.460055 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:26.460170 4809 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 3.
I20260806 04:08:26.461133 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:26.430263 9402 raft_consensus.cc:729] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:26.462245 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:26.462905 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.449795 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:26.459002 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.464102 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:26.449134 9196 consensus_meta.cc:287] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.722s user 0.011s sys 0.001s
I20260806 04:08:26.464800 4931 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:26.471750 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:26.472101 9532 log.cc:924] Time spent T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20260806 04:08:26.473968 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 377) took 1044 ms. Trace:
I20260806 04:08:26.473762 4668 catalog_manager.cc:5495] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:26.475236 4668 catalog_manager.cc:5495] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:26.477507 4768 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:26.452754 4992 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.613s (SENT) [suppressed 18 similar messages]
I20260806 04:08:26.451704 9327 raft_consensus.cc:729] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:26.474378 5037 rpcz_store.cc:276] 0806 04:08:25.429146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.471043 (+ 41897us) service_pool.cc:227] Handling call
0806 04:08:25.471373 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.471720 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.471751 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.482254 (+ 10503us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.056972 (+574718us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:26.128115 (+ 71143us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.276476 (+148361us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.276553 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.473747 (+197194us) raft_consensus.cc:1671] finished
0806 04:08:26.473775 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.473908 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158464,"prepare.queue_time_us":71186,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":3090,"raft.queue_time_us":150654,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":15259,"thread_start_us":499348,"threads_started":1,"wal-append.queue_time_us":720636}
I20260806 04:08:26.501343 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.451139 9510 raft_consensus.cc:686] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:26.499794 4668 catalog_manager.cc:5495] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 3 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260806 04:08:26.535316 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.538407 9402 consensus_queue.cc:260] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.534968 4668 catalog_manager.cc:5495] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:26.539062 9570 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:26.541294 9570 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:26.543129 9157 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.543260 9570 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:26.540680 4668 catalog_manager.cc:5495] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 4, leader changed from d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 4 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:08:26.543676 9157 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.546481 9157 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.542865 5037 raft_consensus.cc:2379] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 6: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:26.543666 4992 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.552325 9510 consensus_queue.cc:237] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:26.553191 9209 consensus_meta.cc:287] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.000s
W20260806 04:08:26.542363 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 64929.9us
I20260806 04:08:26.555066 4992 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.555969 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.556275 4768 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.486045 9303 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:26.557376 5037 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:26.555903 4668 catalog_manager.cc:5495] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 2 to 4, leader changed from d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) to df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65). New cstate: current_term: 4 leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:08:26.562224 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:26.563127 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:26.565470 4992 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.463442 9306 consensus_queue.cc:1035] T 27545c8363cd404189c9e93f3bd1426c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260806 04:08:26.546502 9402 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:26.584476 4992 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.561054 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.556989 9157 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:26.586295 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.586113 4668 catalog_manager.cc:5495] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "22fe557f852d4415a96fd683f970cc24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } }
I20260806 04:08:26.589011 4992 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.589030 5028 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:26.591610 4876 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.562862 9480 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:26.593099 4768 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.567167 9433 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.567512 9275 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:26.585230 9436 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:26.585067 9370 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:26.588251 9450 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:26.637712 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:26.644758 5028 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:26.599188 4768 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.591777 9359 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:08:26.648900 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 199202us
I20260806 04:08:26.652134 4882 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.593569 9157 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:08:26.655870 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 66940.1us
W20260806 04:08:26.655911 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 56847.9us
I20260806 04:08:26.658044 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:26.689112 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.637786 9303 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:26.691282 5867 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:26.592737 9156 consensus_queue.cc:1046] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20260806 04:08:26.640381 9436 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:26.691250 4992 leader_election.cc:398] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:26.695858 4992 leader_election.cc:401] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:08:26.637420 9433 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.687469 4882 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.642632 9207 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.643657 9275 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.693955 4768 leader_election.cc:302] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:26.712924 4768 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:26.693365 9204 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:26.653978 9359 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:26.596342 9209 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:26.688167 9252 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.687992 9293 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:26.655047 9498 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:26.685001 9130 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.688336 9481 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:08:26.693423 9303 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:26.589221 9327 consensus_queue.cc:260] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.694475 9157 raft_consensus.cc:2726] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:08:26.698254 9436 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:26.717712 4808 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [term 10 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:26.712842 9153 raft_consensus.cc:2726] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:26.718348 8533 raft_consensus.cc:2726] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:26.733057 9498 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.718741 9359 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:26.719061 9209 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:26.714169 4768 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:26.701086 4992 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.720112 9340 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:26.756049 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1528) took 1275 ms. Trace:
I20260806 04:08:26.756620 4805 rpcz_store.cc:276] 0806 04:08:25.481155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.919678 (+438523us) service_pool.cc:227] Handling call
0806 04:08:25.942562 (+ 22884us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.942792 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:08:26.375918 (+433126us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.755578 (+379660us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:26.755650 (+ 72us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:26.755725 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.755810 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.755973 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16484,"prepare.queue_time_us":645792,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"thread_start_us":753145,"threads_started":2}
I20260806 04:08:26.754139 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:26.752493 9550 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20260806 04:08:26.744747 9498 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:26.752123 9552 log.cc:924] Time spent T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.296s user 0.000s sys 0.001s
W20260806 04:08:26.751868 9245 consensus_meta.cc:287] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.667s user 0.006s sys 0.006s
I20260806 04:08:26.734498 9303 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:26.752345 9575 log.cc:924] Time spent T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.266s user 0.000s sys 0.001s
W20260806 04:08:26.763899 9563 log.cc:924] Time spent T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260806 04:08:26.715126 9207 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:26.830492 9538 log.cc:924] Time spent T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.510s user 0.000s sys 0.001s
I20260806 04:08:26.831702 9207 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.716981 9275 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.834017 9275 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:26.835960 9543 log.cc:924] Time spent T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.455s user 0.001s sys 0.000s
I20260806 04:08:26.836221 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 380) took 1301 ms. Trace:
I20260806 04:08:26.836664 5030 rpcz_store.cc:276] 0806 04:08:25.535122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.642004 (+106882us) service_pool.cc:227] Handling call
0806 04:08:25.642392 (+ 388us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:25.674905 (+ 32513us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.674933 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:25.675572 (+ 639us) log.cc:842] Serialized 52 byte log entry
0806 04:08:26.127055 (+451483us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:26.490868 (+363813us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.542309 (+ 51441us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.542502 (+ 193us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.542570 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.829698 (+287128us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:26.835996 (+ 6298us) raft_consensus.cc:1671] finished
0806 04:08:26.836025 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.836161 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40709,"prepare.queue_time_us":95237,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":75,"raft.queue_time_us":71786,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":12734,"spinlock_wait_cycles":16771840,"thread_start_us":690918,"threads_started":2,"wal-append.queue_time_us":573136}
W20260806 04:08:26.782291 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 68233us
I20260806 04:08:26.785827 9600 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:26.839164 4768 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:26.752983 9209 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:26.783612 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:26.764719 9535 log.cc:924] Time spent T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.398s user 0.000s sys 0.001s
W20260806 04:08:26.787549 9587 log.cc:924] Time spent T 30e5d6887268432aaf472e51f9ae0444 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 04:08:26.764627 9157 raft_consensus.cc:2726] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:26.787286 9153 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:26.855899 9153 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:26.856407 9153 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:26.788828 4810 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:26.859844 5856 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:08:26.794586 9370 raft_consensus.cc:3032] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:26.860543 9370 raft_consensus.cc:729] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:26.794685 9436 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.839416 9600 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.864905 9436 leader_election.cc:288] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.866124 9600 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.795377 9245 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.867550 8641 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:26.868124 8641 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:26.796227 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 376) took 1440 ms. Trace:
I20260806 04:08:26.869856 5034 rpcz_store.cc:276] 0806 04:08:25.356251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.429875 (+ 73624us) service_pool.cc:227] Handling call
0806 04:08:25.446495 (+ 16620us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:26.230899 (+784404us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.795885 (+564986us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b54000bca54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:26.796149 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1705345280}
I20260806 04:08:26.769698 8532 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:26.871559 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.872915 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:26.874022 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.869462 9245 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:26.876516 9559 log.cc:924] Time spent T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.450s user 0.001s sys 0.000s
I20260806 04:08:26.800045 9130 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:26.878786 9545 log.cc:924] Time spent T a3ae3b855e0d4f98a67493c708f71d33 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.478s user 0.002s sys 0.000s
I20260806 04:08:26.880225 4814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1525) took 1461 ms. Trace:
W20260806 04:08:26.801725 9555 log.cc:924] Time spent T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260806 04:08:26.808549 4804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2435) took 1030 ms. Trace:
I20260806 04:08:26.884788 4804 rpcz_store.cc:276] 0806 04:08:25.777791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.989969 (+212178us) service_pool.cc:227] Handling call
0806 04:08:25.990294 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:26.251636 (+261342us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.808258 (+556622us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400270bd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:26.808478 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592935296}
I20260806 04:08:26.878806 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:26.887285 5866 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:26.815397 4808 raft_consensus.cc:1226] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 04:08:26.891183 4808 raft_consensus.cc:1226] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:26.819790 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 393) took 1056 ms. Trace:
I20260806 04:08:26.764405 9598 consensus_queue.cc:1035] T a0ff8f19c60b43b99aa42dd6c9cee458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.098s
I20260806 04:08:26.895682 5859 rpcz_store.cc:276] 0806 04:08:25.763683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.763843 (+ 160us) service_pool.cc:227] Handling call
0806 04:08:25.783560 (+ 19717us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.798132 (+ 14572us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.798181 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.864300 (+ 66119us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.250302 (+386002us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:26.250348 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.250426 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.819528 (+569102us) raft_consensus.cc:1671] finished
0806 04:08:26.819555 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.819721 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7152,"raft.queue_time_us":26409,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":338853,"threads_started":1,"wal-append.queue_time_us":500089}
W20260806 04:08:26.822471 8180 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.596s user 0.007s sys 0.004s
I20260806 04:08:26.825794 9433 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:26.721851 9204 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.730510 9156 consensus_queue.cc:1046] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260806 04:08:26.725670 9252 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:26.828882 5029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34040 (request call id 659) took 1080 ms. Trace:
W20260806 04:08:26.838726 9529 log.cc:924] Time spent T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.587s user 0.001s sys 0.002s
I20260806 04:08:26.786372 5028 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:26.904459 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:26.764160 9579 log.cc:924] Time spent T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260806 04:08:26.844687 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.761955 9327 raft_consensus.cc:3037] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:26.850381 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:26.849985 9306 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:26.854050 9293 consensus_queue.cc:1046] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:08:26.928516 9360 consensus_queue.cc:1046] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:08:26.929060 9505 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:08:26.800230 9574 log.cc:924] Time spent T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.377s user 0.001s sys 0.001s
I20260806 04:08:26.881059 5857 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:26.880839 4768 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:26.881881 5034 raft_consensus.cc:2379] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 5: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:26.880614 4814 rpcz_store.cc:276] 0806 04:08:25.419176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.846590 (+427414us) service_pool.cc:227] Handling call
0806 04:08:25.861189 (+ 14599us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.861812 (+ 623us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.861844 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.862495 (+ 651us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.428632 (+566137us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:26.428674 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.428742 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.875952 (+447210us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:26.879917 (+ 3965us) raft_consensus.cc:1671] finished
0806 04:08:26.879939 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.880158 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":20705024,"thread_start_us":565694,"threads_started":1,"wal-append.queue_time_us":563145}
I20260806 04:08:26.890992 4924 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:26.827630 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 384) took 1062 ms. Trace:
W20260806 04:08:26.898470 4919 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.045s sys 0.057s
I20260806 04:08:26.898700 4818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 535) took 1500 ms. Trace:
I20260806 04:08:26.861754 9370 consensus_queue.cc:260] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:26.902693 5029 rpcz_store.cc:276] 0806 04:08:25.748022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.934067 (+186045us) service_pool.cc:227] Handling call
0806 04:08:26.828029 (+893962us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540032fad4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98dfaf0 00007f23d9962f2a 00007f23de89fc6e 00007f23d7f96807 00007f23d7f96792 00007f23d7f966c1 00007f23d7f9663c 00007f23d7f958f2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:26.828754 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729322880}
I20260806 04:08:26.917835 5866 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:26.922505 8812 consensus_queue.cc:1035] T a3ae3b855e0d4f98a67493c708f71d33 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:08:26.924404 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:26.925071 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:26.927456 9514 consensus_queue.cc:1046] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:08:26.935668 5036 rpcz_store.cc:276] 0806 04:08:25.765572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.835873 (+ 70301us) service_pool.cc:227] Handling call
0806 04:08:25.836178 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.863132 (+ 26954us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.863158 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.863606 (+ 448us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.251803 (+388197us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:26.251840 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.251917 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.827381 (+575464us) raft_consensus.cc:1671] finished
0806 04:08:26.827411 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.827567 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35325,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":6296,"thread_start_us":330304,"threads_started":1,"wal-append.queue_time_us":512809}
I20260806 04:08:26.935724 4924 raft_consensus.cc:595] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:26.851403 9570 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:26.935853 4919 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.046s sys 0.057s
I20260806 04:08:26.950145 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:26.939044 4995 leader_election.cc:302] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:26.905237 9450 raft_consensus.cc:2726] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:26.933281 5034 raft_consensus.cc:1226] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:26.967335 5034 raft_consensus.cc:1203] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-7.5] Dedup: 3.3->[5.4-7.5]
I20260806 04:08:26.967828 5034 pending_rounds.cc:76] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Aborting all ops after (but not including) 3
I20260806 04:08:26.968050 5034 pending_rounds.cc:97] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260806 04:08:26.968426 5034 raft_consensus.cc:2866] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:26.918409 9433 raft_consensus.cc:3032] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:26.975232 5028 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:26.976490 5029 raft_consensus.cc:1226] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:26.978065 5029 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:26.931816 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 80032.7us
I20260806 04:08:26.991014 4768 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:26.950170 5036 raft_consensus.cc:1226] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:26.880321 9498 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260806 04:08:26.899379 8532 consensus_queue.cc:1035] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260806 04:08:26.935990 4818 rpcz_store.cc:276] 0806 04:08:25.398477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.832038 (+433561us) service_pool.cc:227] Handling call
0806 04:08:25.832347 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.843218 (+ 10871us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.843261 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.843790 (+ 529us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.202958 (+359168us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:26.203005 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.203092 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.878033 (+674941us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:26.881226 (+ 3193us) raft_consensus.cc:1671] finished
0806 04:08:26.881263 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:26.898592 (+ 17329us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4208,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":87864704,"thread_start_us":358815,"threads_started":1,"wal-append.queue_time_us":537423}
I20260806 04:08:27.004676 4817 raft_consensus.cc:1226] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:26.908178 9440 consensus_queue.cc:1046] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260806 04:08:27.008620 5029 raft_consensus.cc:1261] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260806 04:08:27.008531 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1818) took 1020 ms. Trace:
I20260806 04:08:27.011356 5035 rpcz_store.cc:276] 0806 04:08:25.987715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.116716 (+129001us) service_pool.cc:227] Handling call
0806 04:08:26.137376 (+ 20660us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:26.277951 (+140575us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:26.277982 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:26.278408 (+ 426us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.433113 (+154705us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:08:26.433168 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.433275 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.008331 (+575056us) raft_consensus.cc:1671] finished
0806 04:08:27.008352 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.008476 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140199,"raft.queue_time_us":141111,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":154384,"threads_started":1,"wal-append.queue_time_us":148805}
I20260806 04:08:26.941380 9306 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:08:26.948468 9130 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:26.974885 9293 consensus_queue.cc:1225] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260806 04:08:27.014547 4920 raft_consensus.cc:1226] T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 4. Current term is 5. Ops: []
I20260806 04:08:27.016296 5029 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260806 04:08:27.013764 9146 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.007s sys 0.012s
I20260806 04:08:26.961784 8955 raft_consensus.cc:2781] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:26.920143 9196 consensus_queue.cc:1046] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
I20260806 04:08:26.975505 9433 raft_consensus.cc:729] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:27.011711 9370 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:27.026322 9130 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.091059 9209 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.100322 9252 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.112181 9204 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:27.134130 9558 log.cc:924] Time spent T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.710s user 0.004s sys 0.000s
I20260806 04:08:27.141179 9570 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.143386 8532 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:27.148783 9402 consensus_meta.cc:287] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.559s user 0.011s sys 0.002s
I20260806 04:08:27.129652 9303 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:27.171269 9146 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.753s user 0.007s sys 0.012s
I20260806 04:08:27.027966 9371 raft_consensus.cc:3032] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:27.177731 8955 raft_consensus.cc:686] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
W20260806 04:08:27.194913 9322 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.007s sys 0.022s
I20260806 04:08:27.186152 9371 raft_consensus.cc:729] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:27.210883 9322 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s user 0.008s sys 0.023s
I20260806 04:08:27.173290 9359 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:27.254112 9537 log.cc:924] Time spent T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.864s user 0.000s sys 0.009s
I20260806 04:08:27.255862 9443 raft_consensus.cc:3032] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:27.256419 9443 raft_consensus.cc:729] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:27.268640 9562 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.841s user 0.009s sys 0.002s
I20260806 04:08:27.012686 4805 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
W20260806 04:08:27.327785 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 336599us
W20260806 04:08:27.328573 4809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.010s sys 0.017s
W20260806 04:08:27.329243 4809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.011s sys 0.017s
I20260806 04:08:27.346690 4768 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:27.352469 4996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.017s sys 0.046s
W20260806 04:08:27.353273 4996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.017s sys 0.046s
W20260806 04:08:27.354537 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 653304us
I20260806 04:08:27.082598 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 383) took 1322 ms. Trace:
I20260806 04:08:27.362520 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:27.373960 8955 consensus_queue.cc:237] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.182127 8532 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.376708 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.377744 9002 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:27.364718 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:27.365417 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 422784us
I20260806 04:08:27.177722 8641 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.380046 4814 raft_consensus.cc:2379] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:27.381386 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.384565 9130 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.386425 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:27.387714 4809 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:08:27.391640 8532 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.382325 8641 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.395867 4992 leader_election.cc:302] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:27.396831 9303 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.399122 9306 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.399312 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.400179 9306 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.401314 4814 raft_consensus.cc:2379] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:27.402717 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.404290 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b536e5a86f4281a4d9ee87a11d409f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:27.405104 9343 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:27.408241 9359 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.411769 9371 consensus_queue.cc:260] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.209918 9433 consensus_queue.cc:260] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.094691 5032 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1797) took 2373 ms. Trace:
I20260806 04:08:27.370221 9209 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:27.364766 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 460563us
I20260806 04:08:27.181969 8797 raft_consensus.cc:3032] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:08:27.363485 5026 rpcz_store.cc:276] 0806 04:08:25.760375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.787462 (+ 27087us) service_pool.cc:227] Handling call
0806 04:08:25.787826 (+ 364us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.788198 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.788226 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.868807 (+ 80581us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.376251 (+507444us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:26.376294 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.376357 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.801011 (+424654us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:27.027189 (+226178us) raft_consensus.cc:1671] finished
0806 04:08:27.027210 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.082505 (+ 55295us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27007,"raft.queue_time_us":2945,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":20259,"spinlock_wait_cycles":3143040,"thread_start_us":441809,"threads_started":1,"wal-append.queue_time_us":565644}
I20260806 04:08:27.377806 4818 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:27.367111 5038 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1812) took 1600 ms. Trace:
I20260806 04:08:27.405090 4814 raft_consensus.cc:3037] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:08:27.461829 4992 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:27.470263 4813 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
W20260806 04:08:27.471065 9614 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260806 04:08:27.471390 9340 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:27.472174 9340 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:27.475967 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 534168us
I20260806 04:08:27.476709 9481 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:27.478291 9481 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.481468 9481 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.483408 9322 raft_consensus.cc:2726] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:27.489279 9306 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.490379 5037 raft_consensus.cc:2396] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 04:08:27.257815 9443 consensus_queue.cc:260] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.493151 9443 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:27.506920 5856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:27.508388 5856 raft_consensus.cc:2396] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 04:08:27.511976 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:27.502472 8797 raft_consensus.cc:729] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:27.513636 9576 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:27.514571 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:27.465401 5038 rpcz_store.cc:276] 0806 04:08:25.766389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.860755 (+ 94366us) service_pool.cc:227] Handling call
0806 04:08:25.861140 (+ 385us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:25.890340 (+ 29200us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.890378 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:25.891033 (+ 655us) log.cc:842] Serialized 52 byte log entry
0806 04:08:26.294596 (+403563us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:26.309319 (+ 14723us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.312139 (+ 2820us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.312234 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.015309 (+703075us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:27.366113 (+350804us) raft_consensus.cc:1671] finished
0806 04:08:27.366339 (+ 226us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.366795 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15115,"prepare.queue_time_us":11892,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":52,"raft.queue_time_us":42626,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":37414,"spinlock_wait_cycles":25018752,"thread_start_us":357111,"threads_started":1,"wal-append.queue_time_us":521716}
I20260806 04:08:27.516302 8921 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:08:27.516481 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 170102us
I20260806 04:08:27.523536 9433 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:27.514678 9576 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.541867 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:27.542444 9544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.036s sys 0.056s
I20260806 04:08:27.543041 9600 raft_consensus.cc:2726] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:27.544600 9316 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:27.550674 9316 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:27.551050 9357 raft_consensus.cc:3032] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:08:27.551656 9357 raft_consensus.cc:729] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:27.553340 9357 consensus_queue.cc:260] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.515545 5032 rpcz_store.cc:276] 0806 04:08:24.721427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.358999 (+637572us) service_pool.cc:227] Handling call
0806 04:08:25.396568 (+ 37569us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.419571 (+ 23003us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.419622 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.420169 (+ 547us) log.cc:842] Serialized 26 byte log entry
0806 04:08:25.994104 (+573935us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:26.025858 (+ 31754us) log.cc:842] Serialized 14 byte log entry
0806 04:08:26.273515 (+247657us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b1800239ec0. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23d99523d0 00007f23d9964576 00007f23d99750d7 00007f23d9975058
0806 04:08:26.273696 (+ 181us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:26.273772 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:26.837984 (+564212us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:27.094391 (+256407us) raft_consensus.cc:1671] finished
0806 04:08:27.094422 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.094593 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23532,"prepare.queue_time_us":32552,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":266724,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":194,"spinlock_wait_cycles":109332480,"thread_start_us":546570,"threads_started":1,"wal-append.queue_time_us":791137}
I20260806 04:08:27.557971 4816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2429) took 2162 ms. Trace:
I20260806 04:08:27.561422 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2204) took 1084 ms. Trace:
I20260806 04:08:27.558517 4816 rpcz_store.cc:276] 0806 04:08:25.395096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.738358 (+343262us) service_pool.cc:227] Handling call
0806 04:08:25.738671 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:25.761762 (+ 23091us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:25.761809 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:25.762411 (+ 602us) log.cc:842] Serialized 26 byte log entry
0806 04:08:26.318320 (+555909us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:26.318364 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.557004 (+1238640us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b540039a3d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:27.557100 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.557203 (+ 103us) raft_consensus.cc:1671] finished
0806 04:08:27.557221 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.557682 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52445312,"thread_start_us":451150,"threads_started":1,"wal-append.queue_time_us":660427,"wal-append.run_cpu_time_us":13311,"wal-append.run_wall_time_us":1090568}
I20260806 04:08:27.571194 9343 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:27.562031 5037 rpcz_store.cc:276] 0806 04:08:26.477190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.561951 (+ 84761us) service_pool.cc:227] Handling call
0806 04:08:27.561018 (+999067us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b540037d054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:27.561279 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1967378304}
W20260806 04:08:27.577703 9153 consensus_meta.cc:287] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.721s user 0.017s sys 0.001s
I20260806 04:08:27.578348 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 518) took 1704 ms. Trace:
I20260806 04:08:27.578265 9576 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:27.586241 9544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.162s user 0.037s sys 0.056s
I20260806 04:08:27.597384 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:27.578361 9153 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:27.609000 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 210390us
W20260806 04:08:27.609463 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 90920.3us
I20260806 04:08:27.531245 8797 consensus_queue.cc:260] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.612161 8797 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:27.613222 8863 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.614070 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53354 (request call id 546) took 1157 ms. Trace:
I20260806 04:08:27.614645 4813 rpcz_store.cc:276] 0806 04:08:26.456842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.833728 (+376886us) service_pool.cc:227] Handling call
0806 04:08:26.834148 (+ 420us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:27.471003 (+636855us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.613716 (+142713us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54003d8954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:27.614005 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1341222784}
I20260806 04:08:27.616170 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.616319 9196 raft_consensus.cc:3032] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:08:27.616844 9196 raft_consensus.cc:729] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:27.617187 4813 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 10: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:27.620082 4992 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:27.588126 4919 rpcz_store.cc:276] 0806 04:08:25.883867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.885693 (+ 1826us) service_pool.cc:227] Handling call
0806 04:08:25.886104 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:25.886361 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 9
0806 04:08:26.309362 (+423001us) log.cc:842] Serialized 14 byte log entry
0806 04:08:27.577125 (+1267763us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b18002ef720. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23df8d18b1 00007f23df8de661 00007f23d69a94ca 00007f23e008af99 00007f23d686d990 00007f23d6a2f864 00007f23d6a2f155 00007f23d99523d0 00007f23d9964576 00007f23d99750d7 00007f23d9975058
0806 04:08:27.577453 (+ 328us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:27.577585 (+ 132us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:08:27.577715 (+ 130us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.577843 (+ 128us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.578102 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58252,"prepare.queue_time_us":383989,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"spinlock_wait_cycles":8044416,"thread_start_us":1069926,"threads_started":2,"wal-append.queue_time_us":701921}
I20260806 04:08:27.622608 4809 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:27.628882 4805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33564 (request call id 1541) took 1041 ms. Trace:
I20260806 04:08:27.629563 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45410 (request call id 1666) took 1172 ms. Trace:
I20260806 04:08:27.630029 4920 rpcz_store.cc:276] 0806 04:08:26.457240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.457349 (+ 109us) service_pool.cc:227] Handling call
0806 04:08:26.544977 (+ 87628us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:27.025075 (+480098us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.628973 (+603898us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b540004a954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:27.629329 (+ 356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1079477120}
I20260806 04:08:27.632586 9251 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:27.633045 9357 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:27.623342 9306 consensus_queue.cc:1035] T df7a93f69159461280c8b02b8ff62358 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:08:27.633338 9251 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.632581 9156 raft_consensus.cc:2726] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:27.621315 9576 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.643242 4768 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:27.643527 9251 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.643656 9576 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:27.645339 5035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.003s sys 0.012s
W20260806 04:08:27.645767 5035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.003s sys 0.012s
I20260806 04:08:27.629350 4805 rpcz_store.cc:276] 0806 04:08:26.587168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.759063 (+171895us) service_pool.cc:227] Handling call
0806 04:08:27.628477 (+869414us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540039c1d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:27.628661 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":394639104}
I20260806 04:08:27.646871 4818 raft_consensus.cc:1226] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Rejecting Update request from peer 99690415a7b3441fbdb5e943667556a4 for earlier term 3. Current term is 4. Ops: []
I20260806 04:08:27.614035 8863 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.647301 9481 raft_consensus.cc:2726] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 04:08:27.649341 8812 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.651837 8812 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.652424 8863 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.618237 9196 consensus_queue.cc:260] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.653656 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2412) took 2405 ms. Trace:
I20260806 04:08:27.654150 9196 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:27.655606 4876 leader_election.cc:302] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:27.656606 9234 raft_consensus.cc:3032] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:08:27.657191 9234 raft_consensus.cc:729] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:27.657678 4876 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:27.658984 9209 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.656737 9360 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.659541 9209 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.659780 9360 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.660511 9001 raft_consensus.cc:2726] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:27.661253 9001 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 04:08:27.661693 9209 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.662168 9360 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.662822 9234 consensus_queue.cc:260] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.664326 9130 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:27.664987 9130 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.665612 9340 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.666477 9616 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.667339 9616 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.669605 9252 consensus_queue.cc:1046] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:27.670540 9371 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:27.672130 9153 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.672196 9252 raft_consensus.cc:3032] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:27.672832 9252 raft_consensus.cc:729] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:27.673554 8180 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.647042 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.674311 8180 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.675557 4810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33548 (request call id 2413) took 2417 ms. Trace:
I20260806 04:08:27.675932 4810 rpcz_store.cc:276] 0806 04:08:25.257682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.257816 (+ 134us) service_pool.cc:227] Handling call
0806 04:08:25.278311 (+ 20495us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:26.858769 (+1580458us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.675019 (+816250us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54002a2bd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:27.675363 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3473214720}
I20260806 04:08:27.677855 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:27.669590 9616 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:27.674201 9252 consensus_queue.cc:260] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.675063 4805 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:27.682865 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2201) took 1234 ms. Trace:
I20260806 04:08:27.683279 5034 rpcz_store.cc:276] 0806 04:08:26.447784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.966772 (+518988us) service_pool.cc:227] Handling call
0806 04:08:26.967140 (+ 368us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:08:27.019296 (+ 52156us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:08:27.147791 (+128495us) log.cc:842] Serialized 14 byte log entry
0806 04:08:27.482449 (+334658us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:27.483853 (+ 1404us) log.cc:842] Serialized 52 byte log entry
0806 04:08:27.484185 (+ 332us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:27.489596 (+ 5411us) log.cc:842] Serialized 14 byte log entry
0806 04:08:27.491066 (+ 1470us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.491209 (+ 143us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.682369 (+191160us) raft_consensus.cc:1671] finished
0806 04:08:27.682390 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.682570 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6704,"prepare.queue_time_us":133129,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":104,"raft.queue_time_us":16113,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":11103,"thread_start_us":326877,"threads_started":1,"wal-append.queue_time_us":491212}
I20260806 04:08:27.685359 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2211) took 1145 ms. Trace:
I20260806 04:08:27.685698 5028 rpcz_store.cc:276] 0806 04:08:26.539372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:27.023609 (+484237us) service_pool.cc:227] Handling call
0806 04:08:27.121476 (+ 97867us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:27.121953 (+ 477us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:27.121990 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:27.143415 (+ 21425us) log.cc:842] Serialized 52 byte log entry
0806 04:08:27.477305 (+333890us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:08:27.477414 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.477575 (+ 161us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.684973 (+207398us) raft_consensus.cc:1671] finished
0806 04:08:27.684991 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.685191 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5460,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":333095,"threads_started":1,"wal-append.queue_time_us":458508}
I20260806 04:08:27.661834 9001 raft_consensus.cc:979] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24: : Instructing follower df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:08:27.663367 4992 leader_election.cc:302] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:27.691524 4768 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:27.654273 8812 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.654106 4817 rpcz_store.cc:276] 0806 04:08:25.247810 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.249065 (+ 1255us) service_pool.cc:227] Handling call
0806 04:08:25.280584 (+ 31519us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:27.015497 (+1734913us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.653130 (+637633us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54002a35d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:27.653423 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3964332288}
I20260806 04:08:27.716503 9234 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:27.720613 9002 raft_consensus.cc:2726] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:27.712996 9252 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:27.722749 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:27.722968 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 65474.8us
I20260806 04:08:27.727205 4805 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:27.728744 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2200) took 1281 ms. Trace:
W20260806 04:08:27.714319 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 69264.9us
I20260806 04:08:27.729336 4876 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:27.729650 4810 raft_consensus.cc:1226] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 7. Current term is 8. Ops: []
W20260806 04:08:27.738618 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 89596.3us
I20260806 04:08:27.691215 9001 raft_consensus.cc:1067] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Signalling peer df2f1e7811114b8a995f6771fb6a6c8a to start an election
I20260806 04:08:27.744253 8955 raft_consensus.cc:2726] T 8570417a88db412294c21c9108061c30 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:27.744670 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 81496.9us
I20260806 04:08:27.749936 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2212) took 1210 ms. Trace:
I20260806 04:08:27.753489 9130 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.758814 9245 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.767285 8532 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.770028 9576 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:27.772912 9370 consensus_meta.cc:287] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.594s user 0.005s sys 0.010s
I20260806 04:08:27.739466 9153 raft_consensus.cc:2781] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:08:27.773759 9204 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.732959 8180 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.779656 5036 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2206) took 1278 ms. Trace:
I20260806 04:08:27.779775 9600 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:27.808735 9600 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.811151 9600 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:27.729617 5030 rpcz_store.cc:276] 0806 04:08:26.447518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:26.957636 (+510118us) service_pool.cc:227] Handling call
0806 04:08:27.011851 (+ 54215us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:27.012257 (+ 406us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:27.012288 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:27.508332 (+496044us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b2c0001fef8. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d833de11 00007f23d834d442 00007f23d9995c11 00007f23d995a58a 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772
0806 04:08:27.509258 (+ 926us) log.cc:842] Serialized 52 byte log entry
0806 04:08:27.591793 (+ 82535us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:27.591846 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.606030 (+ 14184us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.728261 (+122231us) raft_consensus.cc:1671] finished
0806 04:08:27.728280 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.728554 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":524,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5601,"spinlock_wait_cycles":8685440,"thread_start_us":74951,"threads_started":1,"wal-append.queue_time_us":213984}
I20260806 04:08:27.782990 8921 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:27.783743 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:27.788599 4814 raft_consensus.cc:2454] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:27.786844 5035 rpcz_store.cc:276] 0806 04:08:26.539632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:27.023924 (+484292us) service_pool.cc:227] Handling call
0806 04:08:27.026929 (+ 3005us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:27.027244 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:27.027270 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:27.027701 (+ 431us) log.cc:842] Serialized 26 byte log entry
0806 04:08:27.646260 (+618559us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:27.688338 (+ 42078us) log.cc:842] Serialized 14 byte log entry
0806 04:08:27.743570 (+ 55232us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.743690 (+ 120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.749635 (+ 5945us) raft_consensus.cc:1671] finished
0806 04:08:27.749654 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.749867 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52576,"prepare.queue_time_us":41787,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":55639,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":49052,"thread_start_us":618103,"threads_started":1,"wal-append.queue_time_us":703300}
I20260806 04:08:27.799480 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.799928 9245 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.800868 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:27.800926 8532 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.801800 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:27.802631 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:27.803052 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:27.803714 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:27.817646 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 151037us
I20260806 04:08:27.797601 5036 rpcz_store.cc:276] 0806 04:08:26.501264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:27.005501 (+504237us) service_pool.cc:227] Handling call
0806 04:08:27.006669 (+ 1168us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:27.007134 (+ 465us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:27.007170 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:27.017045 (+ 9875us) log.cc:842] Serialized 52 byte log entry
0806 04:08:27.541130 (+524085us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:27.544201 (+ 3071us) log.cc:842] Serialized 14 byte log entry
0806 04:08:27.545279 (+ 1078us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:27.545425 (+ 146us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:27.779252 (+233827us) raft_consensus.cc:1671] finished
0806 04:08:27.779276 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:27.779471 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":295,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":54,"raft.queue_time_us":3034,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":6767,"thread_start_us":171436,"threads_started":1,"wal-append.queue_time_us":600879}
I20260806 04:08:27.807895 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:27.801898 9204 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.793452 9153 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:27.818759 5034 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:27.818809 9153 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:08:27.821957 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:27.826984 4809 raft_consensus.cc:2396] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:27.827571 5026 raft_consensus.cc:2396] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:08:27.828065 5038 raft_consensus.cc:2454] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:27.831558 8532 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.832721 5854 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260806 04:08:27.835994 5029 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:27.839170 5037 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:27.814098 8921 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.842133 8921 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.842852 9316 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.814604 9001 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:27.851944 9316 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.853133 9245 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:27.842372 5854 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59636 (request call id 732) took 1928 ms (client timeout 2054 ms). Trace:
I20260806 04:08:27.855283 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:27.856724 5857 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:27.848814 8797 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:27.856395 5854 rpcz_store.cc:269] 0806 04:08:25.913743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:25.921105 (+ 7362us) service_pool.cc:227] Handling call
0806 04:08:27.842104 (+1920999us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b5400770954. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:27.842295 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4229628672}
I20260806 04:08:27.863103 9001 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.866324 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:27.867660 5867 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:27.870033 9001 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.873781 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:27.872773 5037 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:27.868044 9234 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:27.878300 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 151655us
I20260806 04:08:27.884654 9443 consensus_queue.cc:1046] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:08:27.885536 9443 consensus_queue.cc:1225] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer df2f1e7811114b8a995f6771fb6a6c8a log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 04:08:27.886081 9443 raft_consensus.cc:3032] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:27.873263 9196 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.871579 9357 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:27.886451 9443 raft_consensus.cc:729] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:27.895907 9450 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:27.897537 8812 consensus_queue.cc:1046] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:08:27.898778 9576 raft_consensus.cc:3032] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:27.900022 4996 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:27.903650 9322 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:27.904264 4808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85"
dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:33548
I20260806 04:08:27.905325 4816 raft_consensus.cc:1226] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 5. Current term is 6. Ops: []
I20260806 04:08:27.922729 4876 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:27.923869 4808 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:08:27.924430 4808 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:27.906104 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.906901 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.907723 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.908527 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.910634 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.911389 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.926761 4805 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:27.913331 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.892423 9204 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.914716 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:27.911324 9196 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.922477 9322 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:27.931557 9322 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:27.931905 4882 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:27.932451 9196 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:27.933074 9196 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:27.933432 9196 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:27.933847 9196 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:27.883598 8797 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.887210 8921 consensus_queue.cc:1035] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
W20260806 04:08:27.941049 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 49568us
I20260806 04:08:27.900882 9576 raft_consensus.cc:729] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:27.943776 9204 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:27.944239 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.944319 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.949069 4809 raft_consensus.cc:2396] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:27.952235 8180 raft_consensus.cc:3037] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:27.950459 9340 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:27.955098 9450 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.955437 9340 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.955737 9450 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.957732 9340 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:27.895162 9234 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:27.963008 8863 consensus_queue.cc:1046] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:08:27.964030 8863 raft_consensus.cc:3032] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:27.964404 8863 raft_consensus.cc:729] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:27.964697 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:27.965586 4920 raft_consensus.cc:2396] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260806 04:08:27.963491 9576 consensus_queue.cc:260] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.897552 9443 consensus_queue.cc:260] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:27.950037 4768 connection.cc:232] RPC call timeout handler was delayed by -1.3518228090833873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20260806 04:08:27.969410 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:27.950266 9204 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:27.977753 9610 consensus_queue.cc:1046] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:08:27.975690 9204 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:27.981026 4768 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:27.957762 9450 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:27.946926 9357 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.987046 4996 leader_election.cc:302] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:27.988445 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:27.989439 4924 raft_consensus.cc:2396] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:27.989889 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:27.990603 5026 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:27.990835 5029 raft_consensus.cc:1261] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:08:27.991032 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:27.992045 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:27.981907 9651 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:27.993319 9156 raft_consensus.cc:2781] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:27.993902 9156 raft_consensus.cc:686] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 5 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:27.992148 9498 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
W20260806 04:08:27.995368 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 104271us
I20260806 04:08:27.992952 9651 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:27.996518 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.999359 9316 raft_consensus.cc:2726] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:08:27.999523 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.000368 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.000756 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:27.971881 8863 consensus_queue.cc:260] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.001467 4817 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:28.001971 8863 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.002642 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.004940 5036 raft_consensus.cc:1261] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:08:28.005096 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:27.982360 9001 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.007321 9156 consensus_queue.cc:237] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.009554 9646 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:27.996599 9402 consensus_queue.cc:1046] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260806 04:08:28.010131 9646 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.011447 5032 raft_consensus.cc:1261] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:08:28.012540 9646 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.013356 4876 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.013127 9498 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.015486 9544 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:28.015633 9576 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:28.015810 9498 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.016208 9544 consensus_queue.cc:1046] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:08:28.017320 9544 raft_consensus.cc:3032] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:08:28.017783 9544 raft_consensus.cc:729] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:28.019323 4927 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:08:28.019800 9544 consensus_queue.cc:260] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.019213 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.010667 9402 raft_consensus.cc:3032] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:08:28.024380 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.029497 4816 raft_consensus.cc:2379] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 4: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:27.972333 9153 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.038926 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:28.034847 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.042430 4814 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 11: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:28.040305 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.043747 4931 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260806 04:08:28.044243 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.059940 4817 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.028671 9402 raft_consensus.cc:729] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:28.061139 4928 raft_consensus.cc:2354] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9.
I20260806 04:08:28.061657 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:28.062049 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.062804 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.063345 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.045141 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.064055 5036 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:28.064960 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.045190 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.065811 4925 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.066900 9443 raft_consensus.cc:3037] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.067277 4879 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.046303 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.066298 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.069535 4931 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:28.069639 8641 raft_consensus.cc:2726] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:28.046516 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.071053 5034 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 9.
I20260806 04:08:28.071341 4927 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 04:08:28.071749 4927 pending_rounds.cc:76] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Aborting all ops after (but not including) 3
I20260806 04:08:28.072010 4927 pending_rounds.cc:97] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260806 04:08:28.072310 4927 raft_consensus.cc:2866] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:28.047559 5026 raft_consensus.cc:1261] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:08:28.074959 8797 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.047658 9501 consensus_queue.cc:1035] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:08:28.075577 9357 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.048777 9592 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:08:28.049033 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:28.075596 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.078255 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:28.079623 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:28.049700 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:28.080256 4928 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:28.051162 5035 raft_consensus.cc:1261] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:08:28.080991 5029 raft_consensus.cc:2379] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:28.051872 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:28.050637 5028 raft_consensus.cc:1261] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:08:28.081794 5857 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:08:28.054399 9245 raft_consensus.cc:3032] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:28.082661 9245 raft_consensus.cc:729] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:28.083657 9245 consensus_queue.cc:260] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.072013 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.050575 9627 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.005s
I20260806 04:08:28.043057 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.086082 4920 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:28.052361 9498 consensus_queue.cc:1046] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:08:28.035432 4924 raft_consensus.cc:1261] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:28.085728 5034 raft_consensus.cc:2454] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:28.035660 9651 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.044385 9153 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.063467 4804 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.080605 9370 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:28.085254 9245 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:28.056507 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24
W20260806 04:08:28.058928 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 89772.8us
I20260806 04:08:28.039534 9234 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:28.113931 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 57544.5us
I20260806 04:08:28.114773 5812 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.115916 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:28.088852 9402 consensus_queue.cc:260] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.093683 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.094552 4882 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.031870 4996 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.102912 5857 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260806 04:08:28.106976 4919 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:28.109742 4925 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:28.110407 4808 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.095288 9204 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.116415 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.118345 4816 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.131426 5812 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:28.125198 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 93426.8us
I20260806 04:08:28.126362 4927 raft_consensus.cc:1226] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 04:08:28.136967 4996 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.127346 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.148228 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.148607 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.144016 4928 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:28.155042 4768 leader_election.cc:398] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:08:28.155414 4768 leader_election.cc:401] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:28.155506 4882 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.123852 8180 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:28.160532 5038 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 9: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:28.118168 8921 raft_consensus.cc:2781] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:28.130337 8532 raft_consensus.cc:2726] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:28.166389 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.126398 9156 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:08:28.171701 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.171767 4882 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.157038 9651 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:28.170115 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:28.177102 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 45783.6us
I20260806 04:08:28.179895 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.117952 9370 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.181943 9544 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.166577 9627 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:28.168972 9316 raft_consensus.cc:2726] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 3. Current term is 4.
I20260806 04:08:28.167616 8921 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:28.196806 4919 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:28.195813 4808 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.173862 9592 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:28.203935 5812 leader_election.cc:302] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24
I20260806 04:08:28.205698 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.206424 4816 raft_consensus.cc:3037] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:28.184584 9651 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:28.209293 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 79773.1us
I20260806 04:08:28.182955 9402 raft_consensus.cc:3037] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:28.185256 9156 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:28.189721 9627 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:28.215627 9417 raft_consensus.cc:2781] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:28.194737 9204 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.200021 8921 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.200189 8180 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:28.200366 9592 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:28.206554 8532 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:28.226173 4876 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:28.227102 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.212715 9651 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.235136 5812 leader_election.cc:302] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.218911 9417 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:28.219795 9627 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:28.244370 9410 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:28.228401 8180 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:28.248322 4876 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.250526 9410 raft_consensus.cc:2726] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:08:28.253575 8812 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:28.259593 9370 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.260763 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.233400 8532 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:28.231448 9592 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.274861 4919 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:28.271046 4931 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 4.
I20260806 04:08:28.243167 9417 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:28.241999 9001 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:28.264961 9371 raft_consensus.cc:2781] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:28.273207 8180 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:08:28.296228 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 48352.7us
I20260806 04:08:28.294226 4876 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.286759 8532 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.234300 9322 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.312223 9612 raft_consensus.cc:3032] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:08:28.312742 9612 raft_consensus.cc:729] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:28.291633 9001 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:28.292630 9370 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:28.319638 5036 raft_consensus.cc:2354] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8.
I20260806 04:08:28.299330 4927 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:28.297075 9501 consensus_queue.cc:1035] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 04:08:28.301082 9357 consensus_queue.cc:1035] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.026s
I20260806 04:08:28.295094 8812 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.342923 8812 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.322835 8533 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:08:28.345774 9322 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:28.360224 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 66105.4us
I20260806 04:08:28.366060 8797 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:08:28.366605 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.367383 4808 raft_consensus.cc:3037] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 8
I20260806 04:08:28.370697 5034 raft_consensus.cc:1261] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:08:28.326416 4927 raft_consensus.cc:595] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.316836 9001 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:28.319028 9371 raft_consensus.cc:686] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:28.383754 9610 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:28.384178 4927 raft_consensus.cc:603] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:28.384344 9610 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.359180 9204 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:28.390066 9316 consensus_queue.cc:1035] T 3ff5845199524dcb8deb5af416122df3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260806 04:08:28.394846 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.420257 4818 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:28.419899 9204 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.395298 4813 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:08:28.395983 9371 consensus_queue.cc:237] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.422088 9204 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.317101 9196 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.423903 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:28.362172 9370 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.425843 9196 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.408020 9156 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:28.426717 5857 raft_consensus.cc:1261] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:08:28.427304 9370 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.426985 9156 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.428206 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.428542 4927 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:28.428898 5032 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:28.429738 9156 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.431669 9610 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.416108 4808 raft_consensus.cc:2454] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
I20260806 04:08:28.448683 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.454003 9245 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:28.467718 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.459257 4816 raft_consensus.cc:2454] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 9.
I20260806 04:08:28.471079 4817 raft_consensus.cc:2354] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 10. Current term is 11.
I20260806 04:08:28.458992 4996 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.462373 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.458038 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.458433 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.482357 4813 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.491815 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.492231 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.492624 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.492717 4924 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 9: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:28.494124 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.494410 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.495075 4931 raft_consensus.cc:2354] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:28.495128 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.496031 4925 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:28.496369 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.497512 4928 raft_consensus.cc:2396] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:08:28.497727 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.499629 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457d26714def4b128d6f433119c45d5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.500716 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.501653 4926 raft_consensus.cc:2379] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:28.502579 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.503856 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.505149 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.505898 4926 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:28.507722 4992 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:28.508737 9600 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:28.509745 4992 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.509045 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.510813 9600 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:28.512619 4931 raft_consensus.cc:2354] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:28.516481 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.517481 4992 leader_election.cc:398] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:28.517601 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.515511 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc3f501bed64af09b04782801118ec3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.520489 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.521606 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.517148 5032 raft_consensus.cc:2454] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:28.518942 5036 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:28.520489 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:28.519037 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.523747 5866 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:28.521164 5030 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:28.529165 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.530689 4926 raft_consensus.cc:2379] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:28.532326 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.482290 9156 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:28.467116 8797 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:28.488849 9370 raft_consensus.cc:2781] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:28.473131 9610 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
W20260806 04:08:28.537586 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 236752us
I20260806 04:08:28.546929 9679 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:28.565675 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.547935 4992 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.548527 8797 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.538991 8955 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:28.550858 9357 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:28.553108 5026 raft_consensus.cc:1226] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: [5.6-9.9]
I20260806 04:08:28.561571 9001 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.536171 4931 raft_consensus.cc:2379] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 9: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:28.528338 9443 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:28.525750 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:28.472952 9245 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.566993 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.459769 9612 consensus_queue.cc:260] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.462906 8532 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.576217 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:28.593361 4768 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:28.537060 9322 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:28.554289 4996 leader_election.cc:302] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.595446 4926 raft_consensus.cc:2379] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 10: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:28.538825 9234 consensus_queue.cc:1035] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260806 04:08:28.604029 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:28.607992 5026 raft_consensus.cc:3037] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.554401 9417 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.610075 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.610306 5034 raft_consensus.cc:2454] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:28.575007 9371 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:28.628319 9371 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.628773 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.612607 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.621973 5029 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.564323 9627 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.609915 9612 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:28.578971 9600 consensus_queue.cc:1035] T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260806 04:08:28.575156 9156 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.646440 4926 raft_consensus.cc:2379] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:28.641793 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.642087 5029 raft_consensus.cc:1226] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 04:08:28.644388 9340 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:28.587998 9357 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.624617 5854 raft_consensus.cc:2354] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 10. Current term is 11.
I20260806 04:08:28.648120 4920 raft_consensus.cc:2454] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:28.600492 8180 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.616307 9443 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.592026 8955 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.663094 9156 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.663329 4927 raft_consensus.cc:595] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.663789 4927 raft_consensus.cc:603] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:28.663831 9156 raft_consensus.cc:2781] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:28.664424 9156 raft_consensus.cc:686] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [term 9 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:28.664539 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.663159 8180 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.665463 4804 raft_consensus.cc:2396] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 04:08:28.665393 9156 consensus_queue.cc:237] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.630515 9371 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.666687 8955 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:28.667014 9576 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.651s user 0.010s sys 0.003s
I20260806 04:08:28.667906 5037 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.640781 8921 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.669370 4919 raft_consensus.cc:2354] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:28.670064 9627 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.670516 8921 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:28.670778 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 122953us
I20260806 04:08:28.670104 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.672027 4818 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.674671 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.675653 4817 raft_consensus.cc:2379] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 8: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:28.574208 9679 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.680260 4925 raft_consensus.cc:2379] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 8: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:28.683005 5036 pending_rounds.cc:76] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Aborting all ops after (but not including) 6
I20260806 04:08:28.683396 5036 pending_rounds.cc:97] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260806 04:08:28.683696 5036 raft_consensus.cc:2866] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:28.684517 4992 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.643163 9204 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:28.694599 9204 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:28.695411 9204 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.663354 9443 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.703673 9679 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.666066 9592 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.705307 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.707152 4804 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:28.716305 5816 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.708531 4992 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.707597 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.718365 4817 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.686451 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.624742 9322 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.659083 9340 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:28.611290 9651 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.722368 4931 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:28.732393 8532 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.734053 8797 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.735538 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.733178 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:28.738514 4804 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.706843 9682 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:28.706596 9592 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.708982 9450 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:28.759730 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:28.763576 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 55126.8us
I20260806 04:08:28.764533 4992 leader_election.cc:302] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.766019 4992 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.749938 9679 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:28.742782 9245 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.743652 9001 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.747742 8955 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:28.751135 4928 raft_consensus.cc:2454] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:28.750180 9340 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.773221 4992 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.740419 9204 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.802629 4813 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.776418 9450 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:28.772006 9679 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:28.789462 9370 raft_consensus.cc:686] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 8 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:28.825313 4882 leader_election.cc:302] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:28.822901 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 268678us
W20260806 04:08:28.829943 9688 log.cc:924] Time spent T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260806 04:08:28.833544 9322 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.834442 9450 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.837152 9450 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.806411 9357 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.844177 9417 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.828279 9679 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:08:28.869220 9678 log.cc:924] Time spent T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.258s user 0.001s sys 0.000s
I20260806 04:08:28.804761 9501 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:28.757076 9682 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.877661 9204 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:28.879698 9651 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:28.879757 5812 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.047s sys 0.071s
W20260806 04:08:28.888643 5812 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.640s user 0.048s sys 0.071s
W20260806 04:08:28.889389 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 654596us
I20260806 04:08:28.890257 5812 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.891160 4804 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:28.871634 9443 raft_consensus.cc:2781] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:28.892985 4817 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:28.903498 4818 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:28.902650 4882 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.895054 9417 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:08:28.916671 9360 raft_consensus.cc:2781] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:28.917303 9360 raft_consensus.cc:686] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:28.917659 4930 raft_consensus.cc:1226] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:28.919086 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.909605 4882 leader_election.cc:302] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.922454 4995 leader_election.cc:398] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 10. Current term is 11.
I20260806 04:08:28.922809 4995 leader_election.cc:401] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:28.925189 4996 leader_election.cc:398] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 10. Current term is 11.
I20260806 04:08:28.903301 9443 raft_consensus.cc:686] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:28.933056 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:28.940912 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.941860 4926 raft_consensus.cc:3037] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:28.945672 5812 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.946808 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.947818 5026 raft_consensus.cc:2454] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:28.949725 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:28.932319 9370 consensus_queue.cc:237] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.929064 9234 raft_consensus.cc:2781] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:28.902626 9501 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:28.918215 9360 consensus_queue.cc:237] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:28.963312 9682 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:28.965024 9501 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:28.960691 9234 raft_consensus.cc:686] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:28.957768 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:28.957530 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 372099us
I20260806 04:08:28.960448 8532 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:28.972518 8532 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:28.974069 4930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "99690415a7b3441fbdb5e943667556a4"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:28.972241 4923 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 9.
I20260806 04:08:28.978695 4930 raft_consensus.cc:595] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Received request to transfer leadership to TS 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:28.979092 4930 raft_consensus.cc:603] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:28.980717 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:28.981459 4817 raft_consensus.cc:2379] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 8: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:28.979697 8532 raft_consensus.cc:3037] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:28.984807 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:28.971292 9443 consensus_queue.cc:237] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.002874 9343 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:08:28.990018 8921 consensus_queue.cc:1046] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260806 04:08:29.006266 8921 raft_consensus.cc:3032] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:29.006634 8921 raft_consensus.cc:729] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:29.007535 8921 consensus_queue.cc:260] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.008145 4931 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:29.000216 9501 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.009783 9501 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.008147 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:28.994515 9682 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:08:29.015085 9682 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:29.015476 9682 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:29.016116 9501 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.009784 5867 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:29.018246 9676 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.439s user 0.004s sys 0.000s
I20260806 04:08:29.025882 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:29.019366 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 190800us
I20260806 04:08:29.032970 4996 leader_election.cc:302] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:28.991597 9234 consensus_queue.cc:237] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.041668 9245 raft_consensus.cc:2726] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:29.042619 9245 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:08:29.045156 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 272017us
I20260806 04:08:29.045393 9322 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:29.045828 9322 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:29.046022 4992 leader_election.cc:398] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:29.046195 9322 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.050832 9450 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.051378 9450 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.026942 4818 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:29.052861 9679 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:29.053444 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 279078us
I20260806 04:08:29.046350 4992 leader_election.cc:401] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:29.054978 9704 raft_consensus.cc:2726] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 5. Current term is 6.
I20260806 04:08:29.056063 4992 leader_election.cc:302] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.053447 9450 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.056113 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:29.053730 9691 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Advancing to term 11
I20260806 04:08:29.059424 4876 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.060818 9544 raft_consensus.cc:2781] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:29.061271 9544 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:29.061663 9544 raft_consensus.cc:3037] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.060534 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:29.062495 5854 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:29.065385 9600 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:29.060580 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.066994 9612 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:29.067493 9612 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:29.067860 9612 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.069357 9694 raft_consensus.cc:3037] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.065945 9600 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.072031 4926 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:29.072472 9600 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.074425 4992 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:29.076238 4992 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:29.076807 9600 raft_consensus.cc:2726] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:29.057331 9450 raft_consensus.cc:2726] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:29.077518 8955 raft_consensus.cc:2726] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:29.079423 4992 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.080663 4992 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.081230 8955 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.081727 9691 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873) with higher term. Message: Invalid argument: T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 10. Current term is 11.
I20260806 04:08:29.083060 4876 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.086489 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:29.090741 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:29.091116 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:29.091950 5853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:29.092656 5853 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:29.088034 9501 raft_consensus.cc:2741] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:08:29.088076 9610 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:29.093914 4879 leader_election.cc:302] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.082412 8921 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.095292 9001 raft_consensus.cc:2726] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:08:29.087797 5867 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 04:08:29.092361 9600 raft_consensus.cc:2781] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:29.095453 4879 leader_election.cc:302] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.096602 9600 raft_consensus.cc:482] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:29.097036 9600 raft_consensus.cc:3037] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.097561 9251 raft_consensus.cc:2726] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:29.087538 4876 leader_election.cc:302] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.101392 4882 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:29.095361 9501 consensus_queue.cc:1046] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
I20260806 04:08:29.107272 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.110893 5812 leader_election.cc:302] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:29.111289 4817 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:29.084520 8532 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.121023 4926 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4"
dest_uuid: "22fe557f852d4415a96fd683f970cc24"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:59506
I20260806 04:08:29.121634 4926 raft_consensus.cc:595] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.121953 4926 raft_consensus.cc:603] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:29.123677 8532 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.124850 4768 leader_election.cc:302] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:29.130375 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.135984 4816 raft_consensus.cc:2379] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:29.111893 5812 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:29.145565 4813 raft_consensus.cc:1261] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:29.145897 4768 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:29.086505 9322 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.147622 4876 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.112599 9251 raft_consensus.cc:2726] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:08:29.149852 4992 leader_election.cc:398] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:29.167133 4992 leader_election.cc:401] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260806 04:08:29.167332 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.168244 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.168938 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.169816 4810 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 9: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:29.124389 9340 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.150900 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.172240 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.172955 4804 raft_consensus.cc:2379] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 10: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:29.153564 9544 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.159718 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.123036 9001 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:29.161962 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.187088 4816 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:29.164075 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.188192 4808 raft_consensus.cc:2354] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
W20260806 04:08:29.165750 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 53941us
I20260806 04:08:29.146260 9627 raft_consensus.cc:3032] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:29.185843 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.166286 8532 raft_consensus.cc:2781] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:29.164870 9156 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.193085 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.172416 9357 raft_consensus.cc:2781] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:29.198477 5029 raft_consensus.cc:3037] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.191066 4879 leader_election.cc:302] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.208969 4817 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:29.211563 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.212610 4813 raft_consensus.cc:2379] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:29.218376 5038 raft_consensus.cc:3037] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.212651 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.187109 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.219044 5026 raft_consensus.cc:3037] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.213020 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.219869 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.220824 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.167603 9679 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.221424 4925 raft_consensus.cc:3037] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:29.227272 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.227394 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.239266 8533 raft_consensus.cc:2726] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:29.226969 4669 catalog_manager.cc:5495] T b5088de56cbc4281b755364a8fddd084 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:08:29.232347 5816 leader_election.cc:302] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:29.240537 4669 catalog_manager.cc:5495] T eaa370887c344934b64e0b3f59c6a3b0 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 1 to 4, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 4 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:29.222589 9357 raft_consensus.cc:482] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:29.242403 9357 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.242475 9371 raft_consensus.cc:2781] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:29.243137 9371 raft_consensus.cc:686] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:29.244114 9371 consensus_queue.cc:237] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.247022 9703 consensus_queue.cc:1035] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.035s
I20260806 04:08:29.192973 9001 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:29.198032 9627 raft_consensus.cc:729] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:29.257524 4768 leader_election.cc:302] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:29.234004 5032 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:29.197908 9316 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 04:08:29.272949 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 81986.7us
I20260806 04:08:29.270547 5026 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
W20260806 04:08:29.275743 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 126032us
I20260806 04:08:29.293844 4669 catalog_manager.cc:5495] T 57d20919f4994e71841cccb9546444a3 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 5 to 7, leader changed from d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68) to 99690415a7b3441fbdb5e943667556a4 (127.2.248.67). New cstate: current_term: 7 leader_uuid: "99690415a7b3441fbdb5e943667556a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:29.294490 4996 leader_election.cc:398] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:29.302904 4996 leader_election.cc:401] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:29.274439 4879 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.305909 4768 leader_election.cc:398] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8.
I20260806 04:08:29.306391 4768 leader_election.cc:401] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:08:29.315862 4996 leader_election.cc:302] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.317067 4996 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.326478 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.327354 4923 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.247793 9610 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:29.256593 9001 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:29.328444 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:29.328469 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 54142.3us
I20260806 04:08:29.329386 4919 raft_consensus.cc:2379] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:29.335007 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.337005 9370 raft_consensus.cc:2726] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:29.337299 9322 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.359277 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.359814 8797 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:29.360033 4930 raft_consensus.cc:3037] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:08:29.360314 8797 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:29.360683 8797 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.338219 9610 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.365299 4931 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:29.368346 8955 raft_consensus.cc:2726] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:29.369020 9685 raft_consensus.cc:2726] T 141037797aa84869bb31e99383874122 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:08:29.368995 4924 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:29.370879 5867 raft_consensus.cc:1226] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:29.372922 8797 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.375164 8797 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:29.311355 9700 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.217321 9156 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.380379 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.382792 5812 leader_election.cc:302] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.387959 8532 raft_consensus.cc:686] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
I20260806 04:08:29.387463 4923 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 7.
I20260806 04:08:29.311103 9002 raft_consensus.cc:2781] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:29.319553 9245 raft_consensus.cc:3037] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.319061 9316 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8.
I20260806 04:08:29.419507 4923 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
W20260806 04:08:29.421011 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 85099.7us
I20260806 04:08:29.329128 9204 raft_consensus.cc:2726] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:08:29.412405 9002 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:29.430088 9002 raft_consensus.cc:3037] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.312426 9691 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20260806 04:08:29.422478 8533 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.432660 8533 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.434849 9343 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:29.438494 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 69291.7us
I20260806 04:08:29.440722 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.420060 4923 pending_rounds.cc:76] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Aborting all ops after (but not including) 8
I20260806 04:08:29.442404 4923 pending_rounds.cc:97] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260806 04:08:29.442785 4923 raft_consensus.cc:2866] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:08:29.445725 4813 raft_consensus.cc:2379] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 8: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:29.434967 8533 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.436373 4925 raft_consensus.cc:2454] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 10.
I20260806 04:08:29.459898 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.495215 5036 raft_consensus.cc:2396] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 04:08:29.492374 8532 consensus_queue.cc:237] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.258042 9702 consensus_queue.cc:1035] T 874b55144315439cb842845d56ccb153 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.169s
I20260806 04:08:29.500916 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.503460 4925 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:29.461215 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4bc22d4f8c4b178aca679682ba4b64" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.509335 4926 raft_consensus.cc:3037] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:29.451129 5863 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
W20260806 04:08:29.511250 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 81874us
I20260806 04:08:29.293349 9612 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.520042 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.512364 4768 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:29.525476 5035 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34042 (request call id 2226) took 1044 ms. Trace:
I20260806 04:08:29.525619 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.526093 5035 rpcz_store.cc:276] 0806 04:08:28.480546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:28.480662 (+ 116us) service_pool.cc:227] Handling call
0806 04:08:29.525154 (+1044492us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400167e54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:29.525371 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1875944320}
I20260806 04:08:29.532871 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:29.534390 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.543815 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.544046 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.544675 4924 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 9.
I20260806 04:08:29.547432 5037 raft_consensus.cc:2396] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260806 04:08:29.550195 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.416776 9417 raft_consensus.cc:2781] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:29.551136 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.552716 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.423012 9501 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.555111 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.553825 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.568224 4931 raft_consensus.cc:2354] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
W20260806 04:08:29.570233 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 253230us
I20260806 04:08:29.621511 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:29.640869 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 128619us
I20260806 04:08:29.649583 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.651252 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.569414 9703 raft_consensus.cc:2726] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:08:29.415759 9700 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:29.663513 9682 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.648s user 0.007s sys 0.010s
I20260806 04:08:29.492503 9576 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.133s
I20260806 04:08:29.664111 9682 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.491971 9343 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.667255 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.668497 4768 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:29.667209 5812 leader_election.cc:302] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:29.683655 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.685177 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.688655 5032 raft_consensus.cc:2379] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 6: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:29.683755 4926 raft_consensus.cc:2454] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:29.425874 9723 consensus_queue.cc:1046] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.264s
I20260806 04:08:29.417198 9627 consensus_queue.cc:260] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.689522 5037 raft_consensus.cc:2379] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:29.693207 4808 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:29.697587 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.698385 4923 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:29.699110 5812 leader_election.cc:302] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:29.697155 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 123577us
I20260806 04:08:29.711237 5812 leader_election.cc:302] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:29.715359 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46963.2us
I20260806 04:08:29.672993 9703 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:29.716276 4930 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 11.
W20260806 04:08:29.688262 9694 consensus_meta.cc:287] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.618s user 0.011s sys 0.000s
I20260806 04:08:29.719111 4930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:29.719883 4930 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:29.695487 9600 consensus_meta.cc:287] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.593s user 0.008s sys 0.004s
I20260806 04:08:29.687671 9544 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.663829 9610 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.727238 4926 raft_consensus.cc:1261] T 457d26714def4b128d6f433119c45d5f P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:08:29.730832 4809 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 04:08:29.694110 9687 raft_consensus.cc:2726] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:08:29.744567 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ac9e0fb0ae490c88fc2d1584718458" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:29.763311 4817 raft_consensus.cc:3037] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 8
I20260806 04:08:29.696208 9417 raft_consensus.cc:686] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [term 10 LEADER]: Becoming Leader. State: Replica: d7c76b6dcd8945cd818b8107caf4a174, State: Running, Role: LEADER
W20260806 04:08:29.711745 9716 log.cc:924] Time spent T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
I20260806 04:08:29.745535 4768 leader_election.cc:398] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9.
I20260806 04:08:29.784915 4768 leader_election.cc:401] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260806 04:08:29.772205 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.715687 8812 raft_consensus.cc:2781] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:08:29.790670 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd13a1778034c599358d37b5d72c374" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:29.792603 4992 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.794243 4992 leader_election.cc:398] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:29.680231 9501 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.716189 9371 raft_consensus.cc:2726] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:29.726806 9340 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Requested vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:29.796567 4992 leader_election.cc:302] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.794134 4930 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
I20260806 04:08:29.730347 9694 raft_consensus.cc:2726] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:29.749207 9723 raft_consensus.cc:3032] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:29.806584 4992 leader_election.cc:302] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.809163 4876 leader_election.cc:302] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.810878 4992 leader_election.cc:302] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:29.748608 9727 raft_consensus.cc:2726] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:29.814342 5037 raft_consensus.cc:1226] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: []
W20260806 04:08:29.818342 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 107198us
I20260806 04:08:29.731900 9703 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.827253 4925 raft_consensus.cc:2454] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 10.
I20260806 04:08:29.731904 9600 raft_consensus.cc:504] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:29.800395 8955 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:29.816066 9340 raft_consensus.cc:2726] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 8. Current term is 9.
I20260806 04:08:29.834731 9360 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:29.808666 9723 raft_consensus.cc:729] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:29.823495 9651 raft_consensus.cc:2781] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:29.823621 9610 raft_consensus.cc:2781] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:29.825191 9322 raft_consensus.cc:2726] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:08:29.832993 9156 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.804983 9702 consensus_queue.cc:1035] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260806 04:08:29.878988 5038 consensus_meta.cc:287] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.660s user 0.007s sys 0.005s
I20260806 04:08:29.822396 4817 raft_consensus.cc:2454] T 50ac9e0fb0ae490c88fc2d1584718458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 8.
I20260806 04:08:29.831326 9450 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:29.845481 9612 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.883508 5038 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:29.846339 8955 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:29.901844 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:29.903923 8955 raft_consensus.cc:3037] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:29.906765 9651 raft_consensus.cc:686] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:29.909722 4814 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:29.913762 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:29.834487 9694 consensus_queue.cc:1035] T 457d26714def4b128d6f433119c45d5f P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20260806 04:08:29.874364 9682 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.874742 9610 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:29.935987 5029 consensus_meta.cc:287] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.713s user 0.012s sys 0.000s
I20260806 04:08:29.936513 5029 raft_consensus.cc:2454] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:29.937778 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:29.882144 9747 raft_consensus.cc:2726] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:29.920218 9343 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.914454 5038 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 6.
I20260806 04:08:29.947742 9245 raft_consensus.cc:2726] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865) with higher term. Message: Invalid argument: T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:29.949910 9754 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:29.941576 9722 raft_consensus.cc:2726] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 04:08:29.941932 9002 consensus_meta.cc:287] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.511s user 0.010s sys 0.000s
I20260806 04:08:29.950489 9723 consensus_queue.cc:260] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:29.953457 9715 log.cc:924] Time spent T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260806 04:08:29.969729 9691 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:29.954035 9720 log.cc:924] Time spent T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.266s user 0.000s sys 0.001s
W20260806 04:08:29.954704 9718 log.cc:924] Time spent T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
I20260806 04:08:29.955657 9700 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:29.956923 9501 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:29.956542 8955 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:29.943373 9357 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.700s user 0.011s sys 0.000s
I20260806 04:08:29.882022 9417 consensus_queue.cc:237] T fdc0072b6de7424caf2520a26bdcd0b0 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.909204 9450 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:29.883741 9360 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:29.900074 9710 log.cc:924] Time spent T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.607s user 0.005s sys 0.005s
I20260806 04:08:30.015384 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.015450 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:30.020925 4879 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:30.022303 5029 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20260806 04:08:30.023898 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 144965us
I20260806 04:08:30.029873 4926 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260806 04:08:29.971312 9726 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260806 04:08:29.939883 9704 raft_consensus.cc:2726] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857) with higher term. Message: Invalid argument: T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for earlier term 6. Current term is 7.
I20260806 04:08:29.940621 9610 raft_consensus.cc:3037] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:30.033737 4818 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
W20260806 04:08:30.033452 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 196926us
W20260806 04:08:30.039127 9739 log.cc:924] Time spent T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260806 04:08:29.907859 9651 consensus_queue.cc:237] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.093035 5032 raft_consensus.cc:1261] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:08:30.040129 4814 raft_consensus.cc:1261] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:30.040654 8812 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:30.051789 5026 raft_consensus.cc:2354] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:30.054744 4882 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:29.963631 9600 leader_election.cc:288] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:30.055621 5036 raft_consensus.cc:1261] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 04:08:29.958673 9703 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:30.062943 4805 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260806 04:08:30.069932 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 259559us
I20260806 04:08:30.109728 4996 leader_election.cc:302] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:30.098974 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 44390us
W20260806 04:08:30.078565 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 57813.4us
W20260806 04:08:30.091142 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 345732us
I20260806 04:08:30.092363 9775 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 04:08:30.117949 4768 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:30.055660 9002 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.126502 4996 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
W20260806 04:08:30.127470 5867 consensus_meta.cc:287] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 0.579s user 0.011s sys 0.001s
I20260806 04:08:30.129521 4876 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:30.088434 9501 raft_consensus.cc:2781] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:30.145550 5867 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 8.
I20260806 04:08:30.070951 9357 raft_consensus.cc:504] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.115172 9645 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:08:30.189152 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 71301.1us
W20260806 04:08:30.177330 9001 consensus_meta.cc:287] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.782s user 0.009s sys 0.005s
I20260806 04:08:30.192765 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
I20260806 04:08:30.072137 9691 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.055814 9754 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.206426 5863 raft_consensus.cc:2379] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a in current term 6: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
W20260806 04:08:30.181640 9710 log.cc:924] Time spent T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260806 04:08:30.105602 8812 raft_consensus.cc:3037] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:08:30.175536 9600 raft_consensus.cc:2781] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:30.228505 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:30.230402 4919 raft_consensus.cc:2379] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 8: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:30.195158 9703 raft_consensus.cc:2726] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:30.204553 9645 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:30.176051 9702 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260806 04:08:30.179834 9685 consensus_queue.cc:1035] T 0bd77d1ae4fc468f8a9d461acb07056b P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260806 04:08:30.264892 9450 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:30.265508 9775 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:30.266000 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 66631.5us
I20260806 04:08:30.254024 5030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34042 (request call id 2236) took 1701 ms. Trace:
I20260806 04:08:30.257745 9357 leader_election.cc:288] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:30.259348 9754 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:30.260123 8955 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:30.247779 9600 raft_consensus.cc:686] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [term 8 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:30.242297 9700 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:30.188916 9699 consensus_queue.cc:1035] T f75362215d584713a892bd7319a22469 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.115s
I20260806 04:08:30.236208 8533 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:30.233387 9752 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:30.244463 9645 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:30.249281 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 741) took 1182 ms. Trace:
I20260806 04:08:30.266132 9775 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.273768 4768 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:30.270821 5030 rpcz_store.cc:276] 0806 04:08:28.552880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:28.660130 (+107250us) service_pool.cc:227] Handling call
0806 04:08:28.703190 (+ 43060us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:28.709444 (+ 6254us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:08:28.710073 (+ 629us) log.cc:842] Serialized 14 byte log entry
0806 04:08:29.088657 (+378584us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:29.089181 (+ 524us) log.cc:842] Serialized 52 byte log entry
0806 04:08:29.089320 (+ 139us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:29.089730 (+ 410us) log.cc:842] Serialized 14 byte log entry
0806 04:08:29.093073 (+ 3343us) log.cc:842] Serialized 14 byte log entry
0806 04:08:29.093469 (+ 396us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:29.093537 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:29.690167 (+596630us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b44000232cc. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d6876f11 00007f23d6944a95 00007f23d69454b2 00007f23d95ed511 00007f23d95e8e1a 00007f23d95dcbee 00007f23d95dbf36 00007f23d95dbb58 00007f23d95e5551 00007f23d95e5509 00007f23d95e5499
0806 04:08:30.253713 (+563546us) raft_consensus.cc:1671] finished
0806 04:08:30.253744 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:30.253940 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107104,"prepare.queue_time_us":639,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":95,"raft.queue_time_us":109060,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":257,"spinlock_wait_cycles":60528512,"thread_start_us":275178,"threads_started":1,"wal-append.queue_time_us":463635}
I20260806 04:08:30.317884 9002 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:30.297279 9592 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:30.332845 9769 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:30.324213 9700 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.333424 9769 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.335089 9700 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:30.335708 9769 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:30.320552 9685 consensus_queue.cc:1035] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260806 04:08:30.337597 9775 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:30.263520 9360 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:30.316915 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 190497us
I20260806 04:08:30.206446 9001 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.266952 9691 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:30.292872 9371 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:30.297748 9450 raft_consensus.cc:2781] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:30.333285 9600 consensus_queue.cc:237] T 50ac9e0fb0ae490c88fc2d1584718458 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.342700 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:30.316084 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 110821us
W20260806 04:08:30.345317 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 194277us
I20260806 04:08:30.347797 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:30.532159 4818 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 04:08:30.532693 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:30.536789 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 409758us
W20260806 04:08:30.542317 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 268601us
I20260806 04:08:30.565696 4768 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:30.544090 4879 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, d7c76b6dcd8945cd818b8107caf4a174; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:30.566979 4816 raft_consensus.cc:2379] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 9: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:30.572057 4817 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
W20260806 04:08:30.592342 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 462867us
I20260806 04:08:30.594527 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.596982 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.597988 4816 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
I20260806 04:08:30.541456 9685 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:30.599504 5026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1837) took 1117 ms. Trace:
I20260806 04:08:30.599937 5026 rpcz_store.cc:276] 0806 04:08:29.458774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.459524 (+ 750us) service_pool.cc:227] Handling call
0806 04:08:30.576047 (+1116523us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b5400166a54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:30.576199 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1248099584}
I20260806 04:08:30.600461 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:30.604947 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:30.605209 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:30.612545 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 46987.7us
I20260806 04:08:30.531390 9450 raft_consensus.cc:482] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:30.618805 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:30.531476 9592 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:30.331233 5856 rpcz_store.cc:276] 0806 04:08:29.059121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.062767 (+ 3646us) service_pool.cc:227] Handling call
0806 04:08:29.063223 (+ 456us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:08:29.063724 (+ 501us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:29.063754 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:08:29.064373 (+ 619us) log.cc:842] Serialized 52 byte log entry
0806 04:08:29.209770 (+145397us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:08:29.209830 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:29.211756 (+ 1926us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:30.227623 (+1015867us) raft_consensus.cc:1671] finished
0806 04:08:30.232750 (+ 5127us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:30.240990 (+ 8240us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1432,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":125085,"threads_started":1,"wal-append.queue_time_us":351922}
I20260806 04:08:30.331729 9752 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.630625 9763 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:08:30.591639 9771 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:30.632858 9771 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.633594 4925 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:30.634212 9694 raft_consensus.cc:482] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:30.634464 9775 raft_consensus.cc:482] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:30.635324 9769 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:30.634866 9694 raft_consensus.cc:504] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.635455 9544 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:30.332033 8533 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.624130 8797 consensus_queue.cc:1035] T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260806 04:08:30.645139 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:30.612190 9685 raft_consensus.cc:686] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:30.645354 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.642370 4669 catalog_manager.cc:5495] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 reported cstate change: term changed from 2 to 3, leader changed from 22fe557f852d4415a96fd683f970cc24 (127.2.248.66) to d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68). New cstate: current_term: 3 leader_uuid: "d7c76b6dcd8945cd818b8107caf4a174" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:08:30.612460 9702 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:08:30.653820 4768 leader_election.cc:302] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:30.648190 9544 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.635991 9747 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:30.646122 9731 raft_consensus.cc:482] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:30.624367 9357 consensus_queue.cc:1046] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 04:08:30.655054 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1701) took 1188 ms. Trace:
I20260806 04:08:30.670868 4919 rpcz_store.cc:276] 0806 04:08:29.459358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.463703 (+ 4345us) service_pool.cc:227] Handling call
0806 04:08:30.647081 (+1183378us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54002e81d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:30.647227 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1617700224}
I20260806 04:08:30.671833 5028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1842) took 1200 ms. Trace:
I20260806 04:08:30.672317 5028 rpcz_store.cc:276] 0806 04:08:29.471523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.680152 (+208629us) service_pool.cc:227] Handling call
0806 04:08:29.704370 (+ 24218us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:29.704644 (+ 274us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:08:30.061839 (+357195us) log.cc:842] Serialized 14 byte log entry
0806 04:08:30.671442 (+609603us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:30.671498 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:08:30.671555 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:30.671629 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:30.671775 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61380,"prepare.queue_time_us":426186,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":71860,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":168,"thread_start_us":868643,"threads_started":3}
I20260806 04:08:30.635111 9775 raft_consensus.cc:504] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.629915 9450 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:30.646296 9699 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:30.654052 9761 consensus_queue.cc:1046] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:08:30.550274 9371 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.605475 9002 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:30.705749 4996 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:30.698694 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:30.702728 4925 raft_consensus.cc:1226] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 04:08:30.695268 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.706796 4816 raft_consensus.cc:2454] T 12fab895c8c34b9d82a95dc9f77980c6 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:30.708768 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:30.635964 9769 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.604295 9501 raft_consensus.cc:686] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [term 8 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:30.632283 9752 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:30.710268 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.741506 4924 raft_consensus.cc:1226] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260806 04:08:30.725046 4996 leader_election.cc:302] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:30.719126 5029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.029s sys 0.098s
W20260806 04:08:30.742429 5029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s user 0.030s sys 0.099s
I20260806 04:08:30.726048 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc0072b6de7424caf2520a26bdcd0b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:30.725535 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:30.726737 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:30.745123 5032 raft_consensus.cc:3037] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:08:30.709767 4817 raft_consensus.cc:1226] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 6. Current term is 7. Ops: []
I20260806 04:08:30.734030 4928 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 04:08:30.725723 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:30.737588 4882 leader_election.cc:302] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:30.737257 4876 leader_election.cc:302] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:30.740074 5030 raft_consensus.cc:1226] T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4 [term 7 LEADER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 04:08:30.741819 5038 raft_consensus.cc:1261] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Refusing update from remote peer df2f1e7811114b8a995f6771fb6a6c8a: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260806 04:08:30.642431 9592 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.750524 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:30.744669 4931 raft_consensus.cc:2454] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 10.
I20260806 04:08:30.748091 4876 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:30.757725 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:30.674753 9360 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260806 04:08:30.758359 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:30.826501 4810 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:30.673063 9731 raft_consensus.cc:504] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.833630 4920 raft_consensus.cc:1226] T 0bd77d1ae4fc468f8a9d461acb07056b P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting Update request from peer df2f1e7811114b8a995f6771fb6a6c8a for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260806 04:08:30.839075 4876 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 91095.1us
I20260806 04:08:30.668900 9747 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:30.843389 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:30.831423 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 94003.6us
I20260806 04:08:30.848358 5026 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:30.758801 9752 raft_consensus.cc:2726] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:30.711553 9699 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:30.755498 9810 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:08:30.877609 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 292874us
I20260806 04:08:30.862329 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:30.720700 9002 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:30.842924 9645 consensus_meta.cc:287] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.510s user 0.010s sys 0.000s
I20260806 04:08:30.958582 9810 raft_consensus.cc:482] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:30.959615 9810 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:30.738054 9795 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260806 04:08:30.958495 4992 leader_election.cc:302] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:30.975292 4931 raft_consensus.cc:1261] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 04:08:30.977325 4919 raft_consensus.cc:1261] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Refusing update from remote peer 99690415a7b3441fbdb5e943667556a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 04:08:30.982275 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:30.844236 9811 raft_consensus.cc:2781] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:08:30.858143 9357 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:08:31.009843 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 117717us
I20260806 04:08:30.856027 9001 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:30.860474 9651 raft_consensus.cc:2726] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:30.842211 9796 consensus_queue.cc:1035] T 0940457704114112b22dab24a18b43b6 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.146s
I20260806 04:08:31.047896 4928 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:30.870886 9763 raft_consensus.cc:686] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [term 8 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:31.064210 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 107374us
W20260806 04:08:31.071741 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 195578us
I20260806 04:08:31.072916 4992 leader_election.cc:302] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:31.074743 4992 leader_election.cc:302] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:31.014835 9811 raft_consensus.cc:482] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:31.018785 9808 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.088299 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:30.970149 9645 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.062624 9694 leader_election.cc:288] T 654e437ca202414286305ce8a8c5c4c2 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:31.059930 9756 raft_consensus.cc:2781] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:31.091809 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1844) took 1619 ms. Trace:
I20260806 04:08:31.065405 9204 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.095589 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:31.087245 9757 raft_consensus.cc:2726] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:31.097514 5037 rpcz_store.cc:276] 0806 04:08:29.472154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.706007 (+233853us) service_pool.cc:227] Handling call
0806 04:08:29.727355 (+ 21348us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:29.827816 (+100461us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:31.091536 (+1263720us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b540040e7d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:31.091751 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":187585792}
I20260806 04:08:31.089293 4919 raft_consensus.cc:2454] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 4.
W20260806 04:08:31.081175 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 415114us
I20260806 04:08:31.104734 4926 raft_consensus.cc:2354] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:30.961144 9685 consensus_queue.cc:237] T 8109ccbed70f46e0932c03eaddc3caf7 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.107302 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:31.082719 9811 raft_consensus.cc:3037] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:31.110134 9824 consensus_queue.cc:1046] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
W20260806 04:08:31.110716 9711 log.cc:924] Time spent T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 1.639s user 0.009s sys 0.006s
W20260806 04:08:31.111101 9725 log.cc:924] Time spent T 28c567d00bdc4e3b800967318816ba4d P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 1.303s user 0.003s sys 0.025s
I20260806 04:08:31.111421 9828 raft_consensus.cc:3032] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:31.111793 9772 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:31.112178 9822 raft_consensus.cc:3032] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Stepping down as leader of term 9
W20260806 04:08:31.112967 9784 log.cc:924] Time spent T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.402s user 0.001s sys 0.000s
I20260806 04:08:31.121526 9757 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.113412 9707 raft_consensus.cc:482] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.114393 9371 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:31.095335 9795 raft_consensus.cc:2781] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:08:31.115622 9699 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:31.116684 9763 consensus_queue.cc:237] T a80ee0f1cc7747e8b736886e63200f37 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.154031 9828 raft_consensus.cc:729] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:31.117987 9831 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:31.118362 9812 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:31.118882 9833 consensus_queue.cc:1035] T 78dcfac52d99420a82f5c7b402a782d7 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.016s
W20260806 04:08:31.119354 9798 log.cc:924] Time spent T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260806 04:08:31.119540 8533 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.120306 9769 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:31.124305 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 57506.5us
I20260806 04:08:31.113205 9791 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:31.202112 9747 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:31.128696 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:31.145816 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:31.105129 9360 raft_consensus.cc:482] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.105767 9544 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:31.059813 9501 consensus_queue.cc:237] T 77a878f4d22e44a290e162038ddd83e1 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.105881 9204 raft_consensus.cc:504] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.154990 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:31.155165 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:31.135273 9806 log.cc:924] Time spent T 21b536e5a86f4281a4d9ee87a11d409f P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.388s user 0.001s sys 0.000s
I20260806 04:08:31.129292 9645 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:31.310333 5032 consensus_meta.cc:287] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.544s user 0.009s sys 0.003s
I20260806 04:08:31.310874 5032 raft_consensus.cc:2454] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 10.
W20260806 04:08:31.113102 9803 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260806 04:08:31.131502 9756 raft_consensus.cc:686] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:31.178575 4816 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:31.144742 9592 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.097469 9808 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.348392 9808 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:31.175454 9761 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.200276 4768 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174
W20260806 04:08:31.420131 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 219976us
I20260806 04:08:31.420930 4768 leader_election.cc:302] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
W20260806 04:08:31.420960 4810 consensus_meta.cc:287] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a: Time spent flushing consensus metadata: real 0.587s user 0.003s sys 0.007s
I20260806 04:08:31.421772 4810 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:31.182324 9795 raft_consensus.cc:482] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:31.165254 9772 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.099509 9775 leader_election.cc:288] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:31.215292 4930 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.002s sys 0.009s
W20260806 04:08:31.434111 4930 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.003s sys 0.009s
I20260806 04:08:31.230279 5867 raft_consensus.cc:2454] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:31.207113 9731 leader_election.cc:288] T 2ea51188c553493cbb9b842b0040bd4f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:31.244174 4813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1587) took 1515 ms. Trace:
I20260806 04:08:31.431330 9795 raft_consensus.cc:3037] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:08:31.295089 4818 raft_consensus.cc:2454] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:31.295818 4814 raft_consensus.cc:3037] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:08:31.298120 9204 leader_election.cc:288] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:31.303351 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 141019us
W20260806 04:08:31.307412 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 85593.1us
I20260806 04:08:31.218822 9002 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.331889 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:31.227610 9610 consensus_meta.cc:287] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.087s user 0.009s sys 0.005s
I20260806 04:08:31.213774 9857 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:31.248222 8812 consensus_meta.cc:287] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174: Time spent flushing consensus metadata: real 1.035s user 0.014s sys 0.001s
I20260806 04:08:31.282924 9360 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.155728 9822 raft_consensus.cc:729] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
W20260806 04:08:31.349537 9782 log.cc:924] Time spent T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.457s user 0.002s sys 0.000s
I20260806 04:08:31.436589 9749 raft_consensus.cc:482] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:31.350613 9836 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.195819 9707 raft_consensus.cc:504] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.239176 9791 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.487665 5028 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:31.488879 9772 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.489398 9791 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.475462 9747 consensus_queue.cc:1046] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260806 04:08:31.229007 9651 raft_consensus.cc:482] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:31.492012 9651 raft_consensus.cc:504] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.461321 9002 raft_consensus.cc:2726] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260806 04:08:31.485363 9749 raft_consensus.cc:504] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.494154 9651 leader_election.cc:288] T 27545c8363cd404189c9e93f3bd1426c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:31.355790 9820 log.cc:924] Time spent T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
I20260806 04:08:31.495074 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:31.495311 9749 leader_election.cc:288] T a3ae3b855e0d4f98a67493c708f71d33 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.350358 9828 consensus_queue.cc:260] T cdc3f501bed64af09b04782801118ec3 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.495390 9707 leader_election.cc:288] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:31.366808 9847 log.cc:924] Time spent T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.237s user 0.001s sys 0.000s
I20260806 04:08:31.348832 9856 raft_consensus.cc:482] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:31.377506 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:31.214223 9831 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.380602 9835 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:31.382408 9810 raft_consensus.cc:504] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:31.385910 9794 log.cc:924] Time spent T 7b4bc22d4f8c4b178aca679682ba4b64 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.675s user 0.000s sys 0.002s
W20260806 04:08:31.390327 9788 log.cc:924] Time spent T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.237s user 0.000s sys 0.000s
I20260806 04:08:31.396778 9771 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.226720 9812 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.229578 9694 raft_consensus.cc:3037] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:31.413360 5813 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.019s sys 0.037s
W20260806 04:08:31.207486 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 482514us
I20260806 04:08:31.422617 4768 leader_election.cc:302] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df2f1e7811114b8a995f6771fb6a6c8a; no voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4
I20260806 04:08:31.436893 4813 rpcz_store.cc:276] 0806 04:08:29.729287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.729435 (+ 148us) service_pool.cc:227] Handling call
0806 04:08:29.729829 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:29.880628 (+150799us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:29.880701 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:30.019193 (+138492us) log.cc:842] Serialized 26 byte log entry
0806 04:08:30.186034 (+166841us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:08:30.186082 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:30.186179 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:31.243876 (+1057697us) raft_consensus.cc:1671] finished
0806 04:08:31.243904 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:31.244091 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":70466,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":21078,"thread_start_us":295233,"threads_started":2,"wal-append.queue_time_us":498840}
I20260806 04:08:31.436406 9808 raft_consensus.cc:482] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.441670 4995 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:31.425431 9766 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.448181 4810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:31.452466 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:31.457275 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:31.178514 9757 raft_consensus.cc:504] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } attrs { replace: true } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.460001 4882 leader_election.cc:302] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:31.462544 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:31.468058 9702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.009s sys 0.006s
I20260806 04:08:31.459218 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:31.472203 4814 raft_consensus.cc:2454] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:31.473855 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:31.350076 5866 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37"
dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174"
mode: GRACEFUL
new_leader_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
from {username='slave'} at 127.0.0.1:53708
I20260806 04:08:31.437373 9848 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.474871 9741 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.483790 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:31.484092 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:31.487193 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 557) took 1686 ms. Trace:
I20260806 04:08:31.487362 9869 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:08:31.458420 9610 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.487591 9836 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.360702 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874b55144315439cb842845d56ccb153" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:31.342409 9844 raft_consensus.cc:482] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:31.495858 4808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33564 (request call id 1596) took 1149 ms. Trace:
I20260806 04:08:31.485661 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:31.479970 9857 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.504766 9685 raft_consensus.cc:482] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.510838 9856 raft_consensus.cc:504] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.516722 5034 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34050 (request call id 1845) took 2044 ms. Trace:
I20260806 04:08:31.526852 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:31.527637 9371 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260806 04:08:31.494544 9450 consensus_meta.cc:287] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 0.784s user 0.009s sys 0.000s
I20260806 04:08:31.487205 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:31.528529 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:31.471140 8812 raft_consensus.cc:504] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.432374 9761 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:31.857295 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 321717us
I20260806 04:08:31.858556 4926 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:31.860643 4920 raft_consensus.cc:2396] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:31.861714 4996 leader_election.cc:302] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:31.864413 5032 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 11.
I20260806 04:08:31.863374 5866 raft_consensus.cc:595] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Received request to transfer leadership to TS df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:31.868477 5866 raft_consensus.cc:603] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:08:31.607385 5038 raft_consensus.cc:2354] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:31.596088 9651 raft_consensus.cc:482] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:31.857903 5034 rpcz_store.cc:276] 0806 04:08:29.472481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.953156 (+480675us) service_pool.cc:227] Handling call
0806 04:08:29.953567 (+ 411us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:29.953954 (+ 387us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:29.953982 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:29.954514 (+ 532us) log.cc:842] Serialized 26 byte log entry
0806 04:08:30.132158 (+177644us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:30.132215 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:30.132309 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:31.516541 (+1384232us) raft_consensus.cc:1671] finished
0806 04:08:31.516559 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:31.516662 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10247,"raft.queue_time_us":82661,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6127,"thread_start_us":177192,"threads_started":1,"wal-append.queue_time_us":365873}
I20260806 04:08:31.871001 4816 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 11.
I20260806 04:08:31.871907 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1720) took 1006 ms. Trace:
I20260806 04:08:31.879138 4920 rpcz_store.cc:276] 0806 04:08:30.865633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.433045 (+567412us) service_pool.cc:227] Handling call
0806 04:08:31.871829 (+438784us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:31.879647 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 419703us
W20260806 04:08:31.603752 4995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 162176us
I20260806 04:08:31.881889 4882 leader_election.cc:302] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:31.865736 4923 rpcz_store.cc:276] 0806 04:08:29.800635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:29.801465 (+ 830us) service_pool.cc:227] Handling call
0806 04:08:29.823937 (+ 22472us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:30.146213 (+322276us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:30.146262 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:30.148258 (+ 1996us) log.cc:842] Serialized 26 byte log entry
0806 04:08:30.591409 (+443151us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:30.591461 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:30.594296 (+ 2835us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:31.486940 (+892644us) raft_consensus.cc:1671] finished
0806 04:08:31.486960 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:31.487126 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9285,"raft.queue_time_us":65509,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":36029,"thread_start_us":703196,"threads_started":2,"wal-append.queue_time_us":578430}
I20260806 04:08:31.732641 9819 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.730362 9812 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.877580 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:31.854748 9771 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.597973 9002 raft_consensus.cc:482] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:31.866348 9702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.156s user 0.011s sys 0.017s
I20260806 04:08:31.875422 4817 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:31.715195 9874 raft_consensus.cc:482] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.887974 5857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:31.859409 4808 rpcz_store.cc:276] 0806 04:08:30.346554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:30.346709 (+ 155us) service_pool.cc:227] Handling call
0806 04:08:30.535528 (+188819us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:30.536011 (+ 483us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:30.536051 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:08:30.607015 (+ 70964us) log.cc:842] Serialized 26 byte log entry
0806 04:08:30.696278 (+ 89263us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:08:30.696335 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:30.696428 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:31.495646 (+799218us) raft_consensus.cc:1671] finished
0806 04:08:31.495667 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:31.495791 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2780,"raft.queue_time_us":3624,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":50446,"thread_start_us":62015,"threads_started":1,"wal-append.queue_time_us":444276}
I20260806 04:08:31.737161 9722 raft_consensus.cc:482] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:31.691095 9791 raft_consensus.cc:482] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.732249 9851 raft_consensus.cc:482] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:31.731587 9360 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:31.733264 9810 leader_election.cc:288] T ebcff471d5ff446ba22c34ec3724dfeb P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:31.859926 9450 raft_consensus.cc:504] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.606627 9808 raft_consensus.cc:504] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.578545 9756 consensus_queue.cc:237] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.603237 9835 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:31.714044 9811 consensus_meta.cc:287] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.570s user 0.012s sys 0.000s
I20260806 04:08:31.888926 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:32.061219 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:32.534508 9811 raft_consensus.cc:504] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.535102 4926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 566) took 1053 ms. Trace:
I20260806 04:08:32.535578 4926 rpcz_store.cc:276] 0806 04:08:31.481548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.870543 (+388995us) service_pool.cc:227] Handling call
0806 04:08:32.535036 (+664493us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:32.536132 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 1.1135e+06us
I20260806 04:08:32.536700 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:31.924903 5035 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
W20260806 04:08:31.978740 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 80000.3us
I20260806 04:08:32.538051 4925 raft_consensus.cc:2379] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 8: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:31.870265 9741 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.883641 9749 raft_consensus.cc:2781] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:08:31.979941 4818 raft_consensus.cc:1261] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:08:31.659600 8955 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:08:31.988335 4882 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 98000.1us
I20260806 04:08:32.558643 4817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53354 (request call id 599) took 1093 ms. Trace:
I20260806 04:08:32.559183 4817 rpcz_store.cc:276] 0806 04:08:31.448328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.452100 (+ 3772us) service_pool.cc:227] Handling call
0806 04:08:32.542058 (+1089958us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:08:32.040050 4992 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.2.248.66:34857 timed out after 1.630s (SENT) [suppressed 2 similar messages]
I20260806 04:08:32.555013 5857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45574 (request call id 426) took 1026 ms. Trace:
I20260806 04:08:32.059515 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:32.035437 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:31.925395 5863 raft_consensus.cc:2396] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:08:32.539608 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:32.540503 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:32.540539 4927 consensus_meta.cc:287] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 1.010s user 0.011s sys 0.000s
I20260806 04:08:32.538621 4925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1717) took 1673 ms. Trace:
I20260806 04:08:32.541764 4996 leader_election.cc:302] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:31.880223 9763 raft_consensus.cc:482] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:31.877162 9245 raft_consensus.cc:482] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:31.880414 9815 consensus_queue.cc:1035] T 086f8af9433f44aa8aa4fccbfa90eb02 P d7c76b6dcd8945cd818b8107caf4a174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.730s
I20260806 04:08:31.884312 9707 consensus_queue.cc:1035] T f573a65ecea34309b58a81b5fcbfdc02 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.726s
I20260806 04:08:31.884155 9370 consensus_queue.cc:1035] T 0ca2ae1bef38445b908545a06111f237 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.726s
I20260806 04:08:31.881033 9769 raft_consensus.cc:2781] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:08:31.979821 9747 raft_consensus.cc:2781] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:32.038596 9612 raft_consensus.cc:482] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.196064 3041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 04:08:32.561515 5857 rpcz_store.cc:276] 0806 04:08:31.528313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.870395 (+342082us) service_pool.cc:227] Handling call
0806 04:08:32.554944 (+684549us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:32.656788 4927 raft_consensus.cc:2454] T 9b0f147300804e5fab336f0242e01ec4 P 22fe557f852d4415a96fd683f970cc24 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 9.
W20260806 04:08:32.275089 9882 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260806 04:08:32.541040 9775 raft_consensus.cc:482] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
W20260806 04:08:32.612991 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 71317.4us
I20260806 04:08:32.674746 9775 raft_consensus.cc:504] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:32.676312 4651 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 997e4d9ff3ca465199a888e8af23ed7b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 99690415a7b3441fbdb5e943667556a4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } } }
I20260806 04:08:32.677472 9775 leader_election.cc:288] T b75ec76cc4cd43bc90264007dff133f3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:32.281219 9836 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.678967 9836 raft_consensus.cc:482] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.558153 9771 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.679503 9836 raft_consensus.cc:504] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.680308 9812 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:32.680826 9812 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.681486 9771 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.557976 9874 raft_consensus.cc:504] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.684325 9874 leader_election.cc:288] T 9b0f147300804e5fab336f0242e01ec4 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.684995 9874 raft_consensus.cc:2726] T da2e93c822734c02a26703dabb801932 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:32.665095 9769 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:32.685634 9769 raft_consensus.cc:3037] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:32.685700 9874 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.686676 9812 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.218921 9204 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.665125 9245 raft_consensus.cc:504] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.687455 9204 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.689016 9245 leader_election.cc:288] T 141037797aa84869bb31e99383874122 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.670102 9747 raft_consensus.cc:482] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:32.689318 9204 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:32.641459 5859 raft_consensus.cc:1226] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 04:08:32.689711 9747 raft_consensus.cc:3037] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:32.689607 9612 raft_consensus.cc:504] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.691879 9612 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.692358 9612 raft_consensus.cc:482] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:32.692903 5859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45574 (request call id 423) took 1217 ms. Trace:
I20260806 04:08:32.692838 9612 raft_consensus.cc:504] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.693308 5859 rpcz_store.cc:276] 0806 04:08:31.475276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.475444 (+ 168us) service_pool.cc:227] Handling call
0806 04:08:31.486777 (+ 11333us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:08:32.690018 (+1203241us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:32.692626 (+ 2608us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540042d1d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:32.692835 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2654590336}
I20260806 04:08:32.356228 9761 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.697324 9811 leader_election.cc:288] T 12fab895c8c34b9d82a95dc9f77980c6 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 6 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.697624 9761 consensus_queue.cc:1046] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:08:32.357404 9869 raft_consensus.cc:2726] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:08:32.023375 9831 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.551934 9360 raft_consensus.cc:482] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:32.064568 9877 raft_consensus.cc:482] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
W20260806 04:08:32.672333 4927 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1719) took 1807 ms (client timeout 2184 ms). Trace:
I20260806 04:08:31.884512 9651 raft_consensus.cc:504] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.681555 9836 leader_election.cc:288] T 0ca2ae1bef38445b908545a06111f237 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.560424 9002 raft_consensus.cc:504] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.696403 9707 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.736004 5029 raft_consensus.cc:1226] T 228767bbb3b643c9977ce6fc5b074827 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Rejecting Update request from peer 22fe557f852d4415a96fd683f970cc24 for earlier term 9. Current term is 10. Ops: []
I20260806 04:08:32.552610 8955 raft_consensus.cc:482] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:32.543596 9702 raft_consensus.cc:686] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:32.662408 4925 rpcz_store.cc:276] 0806 04:08:30.864755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.877198 (+1012443us) service_pool.cc:227] Handling call
0806 04:08:32.538562 (+661364us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:31.878996 9766 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:31.863561 9685 raft_consensus.cc:504] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.698710 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:32.795143 5026 raft_consensus.cc:2454] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 5.
I20260806 04:08:32.263688 9817 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.351611 9757 leader_election.cc:288] T 2e4a7d4c7807476384d5ce509d4029f9 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.283821 9857 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.349591 9856 leader_election.cc:288] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
W20260806 04:08:32.721324 4927 rpcz_store.cc:269] 0806 04:08:30.865353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.377154 (+511801us) service_pool.cc:227] Handling call
0806 04:08:32.672266 (+1295112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:32.865192 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:32.668332 5867 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 11.
I20260806 04:08:32.613489 9865 raft_consensus.cc:2726] T 57d20919f4994e71841cccb9546444a3 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:08:32.722152 9703 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.959s user 0.047s sys 0.119s
I20260806 04:08:32.740566 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54785970285b4054b83ec528e156c25c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:32.781563 9869 raft_consensus.cc:482] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:32.744004 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:32.586050 9851 raft_consensus.cc:504] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.748054 9612 leader_election.cc:288] T faa361eb5ab5463895c3f7af5e87237c P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:32.748163 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01722f8c013b4c0fb8ae76f83562ca5f" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:32.895026 5036 raft_consensus.cc:2379] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 8: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:32.804735 9836 raft_consensus.cc:3032] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:08:32.750006 5032 raft_consensus.cc:1261] T 7b1f802648ed4dccb7235c747e9b1553 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:08:32.667358 9610 leader_election.cc:288] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:31.869371 9844 raft_consensus.cc:504] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.742635 9741 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.686406 9874 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:31.978147 9819 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.785257 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:32.780347 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:32.894181 9703 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.137s user 0.047s sys 0.119s
I20260806 04:08:32.903007 5854 raft_consensus.cc:2379] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate d7c76b6dcd8945cd818b8107caf4a174 in this term.
I20260806 04:08:32.558511 9791 raft_consensus.cc:504] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:32.916313 4992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.66:34857, user_credentials={real_user=slave}} blocked reactor thread for 876221us
I20260806 04:08:32.749167 8955 raft_consensus.cc:3037] T 589278a46368433c87a150021544ba2e P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:32.637121 9763 raft_consensus.cc:504] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.921470 5029 raft_consensus.cc:2454] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:32.898331 9836 raft_consensus.cc:729] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:31.869596 9848 raft_consensus.cc:504] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.880358 8812 leader_election.cc:288] T 5666124a588f45948039fcd27abc9cf4 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 4 election: Requested vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.887241 4927 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:32.928645 4992 leader_election.cc:302] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:32.560271 9722 raft_consensus.cc:504] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.892222 4923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1725) took 1016 ms. Trace:
I20260806 04:08:32.929723 4923 rpcz_store.cc:276] 0806 04:08:31.875994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:32.709212 (+833218us) service_pool.cc:227] Handling call
0806 04:08:32.892129 (+182917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:32.749760 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
W20260806 04:08:32.897738 5813 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.007s sys 0.045s
W20260806 04:08:32.931290 5813 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.826s user 0.007s sys 0.045s
I20260806 04:08:32.736657 9360 raft_consensus.cc:504] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.931376 5866 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
W20260806 04:08:32.893173 9694 consensus_meta.cc:287] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.282s user 0.013s sys 0.000s
I20260806 04:08:32.749038 9707 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:32.736641 9877 raft_consensus.cc:504] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:32.791153 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd77d1ae4fc468f8a9d461acb07056b" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
W20260806 04:08:32.908634 5863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59650 (request call id 748) took 2335 ms (client timeout 2215 ms). Trace:
I20260806 04:08:32.262609 8797 raft_consensus.cc:3032] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:32.913439 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:32.918817 5816 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.016s sys 0.028s
W20260806 04:08:32.924165 4876 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.005s sys 0.013s
W20260806 04:08:32.954470 5816 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s user 0.017s sys 0.029s
I20260806 04:08:32.893834 9757 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:32.784298 9808 leader_election.cc:288] T 793f3bf4a49a4d108609efd686fb6c85 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:32.917940 9795 consensus_meta.cc:287] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4: Time spent flushing consensus metadata: real 1.479s user 0.014s sys 0.000s
W20260806 04:08:32.939842 9443 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.006s sys 0.017s
I20260806 04:08:32.963069 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
W20260806 04:08:32.942301 5863 rpcz_store.cc:269] 0806 04:08:30.565921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:30.566057 (+ 136us) service_pool.cc:227] Handling call
0806 04:08:32.901278 (+2335221us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001a2054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:32.901449 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3028647680}
I20260806 04:08:32.899266 9869 raft_consensus.cc:504] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:32.954524 4876 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.917s user 0.005s sys 0.013s
I20260806 04:08:32.954535 5032 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
W20260806 04:08:32.998533 4770 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.032s sys 0.075s
W20260806 04:08:32.998992 4770 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.023s user 0.033s sys 0.075s
I20260806 04:08:33.000464 5034 raft_consensus.cc:1261] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Refusing update from remote peer 22fe557f852d4415a96fd683f970cc24: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:08:32.928416 9450 leader_election.cc:288] T a4bb640df6c2435ea743b7157dd130bd P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:32.940050 9775 raft_consensus.cc:2726] T 874b55144315439cb842845d56ccb153 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:32.967617 5037 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34146 (request call id 409) took 1483 ms. Trace:
W20260806 04:08:32.961352 9443 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.007s sys 0.018s
W20260806 04:08:32.982808 4993 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.013s sys 0.033s
W20260806 04:08:33.013322 4993 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.014s sys 0.034s
I20260806 04:08:32.987365 9002 leader_election.cc:288] T 8570417a88db412294c21c9108061c30 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.961678 8812 raft_consensus.cc:482] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:32.991499 9360 leader_election.cc:288] T da2e93c822734c02a26703dabb801932 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:32.963979 4927 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 11.
I20260806 04:08:33.031407 4927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55924 (request call id 568) took 1523 ms. Trace:
I20260806 04:08:33.031980 4927 rpcz_store.cc:276] 0806 04:08:31.508083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:32.937288 (+1429205us) service_pool.cc:227] Handling call
0806 04:08:33.031321 (+ 94033us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:32.960166 4923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1ae23ce1104b3688f3f881c64ed473" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.003841 9865 raft_consensus.cc:482] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:33.005803 4992 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4; no voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:32.953811 9694 raft_consensus.cc:2726] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:08:33.014791 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.043599 4992 leader_election.cc:302] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:33.017060 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.045436 4920 raft_consensus.cc:2454] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:33.046057 4920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1728) took 1163 ms. Trace:
I20260806 04:08:33.046458 4920 rpcz_store.cc:276] 0806 04:08:31.883071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:32.966269 (+1083198us) service_pool.cc:227] Handling call
0806 04:08:33.045996 (+ 79727us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:33.018638 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.048635 4924 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P 22fe557f852d4415a96fd683f970cc24 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 11.
I20260806 04:08:33.049489 4924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1730) took 1164 ms. Trace:
I20260806 04:08:32.955463 8797 raft_consensus.cc:729] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:33.020226 9245 raft_consensus.cc:482] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.026014 9610 raft_consensus.cc:482] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:33.007182 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.026259 9879 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.026463 9450 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.026944 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.077883 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.027822 9877 leader_election.cc:288] T 57d20919f4994e71841cccb9546444a3 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.012100 9703 raft_consensus.cc:482] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.028616 9874 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.028874 9885 raft_consensus.cc:482] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.028620 5867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.051402 9204 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:08:32.924475 9702 consensus_queue.cc:237] T 96c2cb602ae44188a8a52e2cecd5a807 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.049885 4924 rpcz_store.cc:276] 0806 04:08:31.885277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:32.968072 (+1082795us) service_pool.cc:227] Handling call
0806 04:08:33.049424 (+ 81352us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:33.058429 5030 raft_consensus.cc:2396] T 0bd77d1ae4fc468f8a9d461acb07056b P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260806 04:08:33.046244 9600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.927s user 0.009s sys 0.036s
I20260806 04:08:33.020751 9612 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.027125 9707 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:33.079368 4769 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.270s user 0.038s sys 0.058s
W20260806 04:08:33.096865 4769 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.498s user 0.038s sys 0.058s
I20260806 04:08:33.038061 9811 raft_consensus.cc:482] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:33.051656 9800 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.018s sys 0.022s
I20260806 04:08:33.043036 9775 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:33.037891 9741 raft_consensus.cc:482] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df2f1e7811114b8a995f6771fb6a6c8a)
I20260806 04:08:33.107013 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75362215d584713a892bd7319a22469" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.084781 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:32.889786 9817 raft_consensus.cc:504] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.085631 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.044874 9360 raft_consensus.cc:482] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:08:33.086377 4768 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.020s sys 0.031s
W20260806 04:08:33.038280 4879 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.020s sys 0.042s
W20260806 04:08:33.110735 4879 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.921s user 0.020s sys 0.042s
I20260806 04:08:33.039077 8812 raft_consensus.cc:504] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:33.111109 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 921433us
W20260806 04:08:33.090197 4882 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.020s sys 0.051s
I20260806 04:08:33.008504 5037 rpcz_store.cc:276] 0806 04:08:31.483982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.484124 (+ 142us) service_pool.cc:227] Handling call
0806 04:08:31.491151 (+ 7027us) raft_consensus.cc:1457] Updating replica for 4 ops
0806 04:08:31.491608 (+ 457us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:31.491637 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 4 ops
0806 04:08:31.492440 (+ 803us) log.cc:842] Serialized 104 byte log entry
0806 04:08:31.779033 (+286593us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:08:32.545454 (+766421us) log.cc:842] Serialized 14 byte log entry
0806 04:08:32.554179 (+ 8725us) log.cc:842] Serialized 14 byte log entry
0806 04:08:32.554515 (+ 336us) log.cc:842] Serialized 14 byte log entry
0806 04:08:32.554741 (+ 226us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:32.554939 (+ 198us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:08:32.967292 (+412353us) raft_consensus.cc:1671] finished
0806 04:08:32.967323 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:32.967538 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8136,"prepare.queue_time_us":1894092,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":3591,"raft.queue_time_us":20108,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":191988,"spinlock_wait_cycles":77179904,"thread_start_us":635931,"threads_started":2,"wal-append.queue_time_us":393564}
W20260806 04:08:33.112444 4768 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.021s sys 0.031s
W20260806 04:08:33.112670 4882 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.021s sys 0.052s
I20260806 04:08:32.997987 9795 raft_consensus.cc:504] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:33.113194 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 576067us
W20260806 04:08:33.059751 9251 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.054s sys 0.080s
W20260806 04:08:33.117821 9251 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.919s user 0.054s sys 0.080s
I20260806 04:08:33.120085 9251 consensus_queue.cc:1046] T 228767bbb3b643c9977ce6fc5b074827 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 04:08:33.095436 4928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45410 (request call id 1713) took 2226 ms (client timeout 1630 ms). Trace:
I20260806 04:08:33.120299 9741 raft_consensus.cc:504] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:33.120944 4928 rpcz_store.cc:269] 0806 04:08:30.861867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:30.861999 (+ 132us) service_pool.cc:227] Handling call
0806 04:08:33.088037 (+2226038us) spinlock_profiling.cc:243] Waited 72.8 ms on lock 0x7b54004141d4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:33.088259 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":167326976}
I20260806 04:08:33.121354 9769 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.122058 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:33.122638 4928 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:33.122956 9890 raft_consensus.cc:482] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.123361 9627 raft_consensus.cc:3037] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:33.123551 9890 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.124898 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.125551 5029 raft_consensus.cc:2454] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:33.038339 9771 raft_consensus.cc:482] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.125713 9890 leader_election.cc:288] T 98064c6d64f74659bfa102bd5a341635 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.126164 9771 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.127081 9851 leader_election.cc:288] T 5854d52dcd124250a917b57b14dd2afc P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:33.096500 9600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.978s user 0.010s sys 0.036s
I20260806 04:08:33.128118 9775 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.127455 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.129163 9771 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.129683 5029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf27da0cc45d4501b4dac0fbb6fedc16" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:33.131338 9763 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:33.132040 5026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 407) took 1648 ms (client timeout 1756 ms). Trace:
I20260806 04:08:33.132578 9791 leader_election.cc:288] T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.133266 4768 leader_election.cc:398] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Vote denied by peer 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701) with higher term. Message: Invalid argument: T 4fef94f36fa34d64a4485e7136f39f85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df2f1e7811114b8a995f6771fb6a6c8a for earlier term 6. Current term is 7.
I20260806 04:08:33.133591 4768 leader_election.cc:401] T 4fef94f36fa34d64a4485e7136f39f85 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260806 04:08:33.133828 9819 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.107528 9703 raft_consensus.cc:504] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } attrs { replace: true } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.135702 9884 consensus_queue.cc:1035] T 54785970285b4054b83ec528e156c25c P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:08:33.135915 9877 consensus_queue.cc:1046] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:08:33.136704 9877 consensus_queue.cc:1225] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Peer 99690415a7b3441fbdb5e943667556a4 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 04:08:33.098414 4876 leader_election.cc:302] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a
I20260806 04:08:33.136988 9703 leader_election.cc:288] T eaa370887c344934b64e0b3f59c6a3b0 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.137142 9884 raft_consensus.cc:3032] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:08:33.137548 9884 raft_consensus.cc:729] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:33.137676 9763 raft_consensus.cc:2726] T 01722f8c013b4c0fb8ae76f83562ca5f P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260806 04:08:33.107775 9800 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.592s user 0.019s sys 0.022s
I20260806 04:08:33.138484 9884 consensus_queue.cc:260] T 7b4bc22d4f8c4b178aca679682ba4b64 P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
W20260806 04:08:33.099962 4878 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.895s user 0.024s sys 0.029s
I20260806 04:08:33.095031 9885 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.100327 9844 leader_election.cc:288] T df7a93f69159461280c8b02b8ff62358 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:33.081769 9592 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.020s sys 0.045s
I20260806 04:08:33.019762 9757 raft_consensus.cc:504] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.079144 9857 raft_consensus.cc:482] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.055761 9812 raft_consensus.cc:2781] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:08:33.117282 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.139974 4876 leader_election.cc:302] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4; no voters: df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:33.140417 4878 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.941s user 0.024s sys 0.029s
I20260806 04:08:33.093590 9443 raft_consensus.cc:482] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.122915 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.083853 9879 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.094719 9766 leader_election.cc:288] T cdc3f501bed64af09b04782801118ec3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.139215 9763 consensus_queue.cc:1035] T 7b1f802648ed4dccb7235c747e9b1553 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260806 04:08:33.154544 9651 leader_election.cc:288] T 7659c18c5db84b729be8dce3bcb9dd32 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.096850 9749 raft_consensus.cc:686] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [term 5 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:33.155609 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.141374 4768 leader_election.cc:302] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:33.122418 9741 leader_election.cc:288] T 874b55144315439cb842845d56ccb153 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.141309 4928 raft_consensus.cc:2454] T a4bb640df6c2435ea743b7157dd130bd P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:33.072863 9245 raft_consensus.cc:504] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.161484 5037 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 4.
I20260806 04:08:33.184690 4768 leader_election.cc:302] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, df2f1e7811114b8a995f6771fb6a6c8a; no voters: d7c76b6dcd8945cd818b8107caf4a174
I20260806 04:08:33.184945 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.162868 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
W20260806 04:08:33.163395 5035 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34146 (request call id 406) took 1680 ms (client timeout 1726 ms). Trace:
W20260806 04:08:33.187676 5035 rpcz_store.cc:269] 0806 04:08:31.483126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.483274 (+ 148us) service_pool.cc:227] Handling call
0806 04:08:33.163239 (+1679965us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540008e654. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:33.163338 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986707840}
W20260806 04:08:33.132431 5026 rpcz_store.cc:269] 0806 04:08:31.483436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:31.483533 (+ 97us) service_pool.cc:227] Handling call
0806 04:08:33.131879 (+1648346us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b540037d054. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:33.131982 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2132424064}
I20260806 04:08:33.164343 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.167351 5862 raft_consensus.cc:2454] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 8.
I20260806 04:08:33.171712 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a7d4c7807476384d5ce509d4029f9" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.189589 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.189761 4928 raft_consensus.cc:2454] T 2e4a7d4c7807476384d5ce509d4029f9 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:33.022428 9836 consensus_queue.cc:260] T 01722f8c013b4c0fb8ae76f83562ca5f P df2f1e7811114b8a995f6771fb6a6c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.171974 5028 raft_consensus.cc:2379] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22fe557f852d4415a96fd683f970cc24 in current term 11: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:33.091085 9685 leader_election.cc:288] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.077298 9610 raft_consensus.cc:504] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.172331 4808 raft_consensus.cc:1261] T f75362215d584713a892bd7319a22469 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Refusing update from remote peer d7c76b6dcd8945cd818b8107caf4a174: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 04:08:33.193040 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.193250 9722 leader_election.cc:288] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
W20260806 04:08:33.163364 9592 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.804s user 0.020s sys 0.045s
I20260806 04:08:33.194371 4882 leader_election.cc:302] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:33.130532 9775 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
W20260806 04:08:33.215101 4996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 69838us
I20260806 04:08:33.221530 5029 raft_consensus.cc:3037] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:08:33.196014 4928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 563) took 1715 ms. Trace:
I20260806 04:08:33.053202 9865 raft_consensus.cc:504] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.088341 9450 raft_consensus.cc:504] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } attrs { replace: true } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.158602 9848 leader_election.cc:288] T 589278a46368433c87a150021544ba2e P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.246412 9865 leader_election.cc:288] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.247234 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2e93c822734c02a26703dabb801932" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.247761 5032 raft_consensus.cc:2454] T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 10.
I20260806 04:08:33.248342 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51188c553493cbb9b842b0040bd4f" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:33.229813 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 45249.2us
I20260806 04:08:33.249526 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654e437ca202414286305ce8a8c5c4c2" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.108027 9811 raft_consensus.cc:504] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.234067 4814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228767bbb3b643c9977ce6fc5b074827" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.235071 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8109ccbed70f46e0932c03eaddc3caf7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.171940 9600 raft_consensus.cc:482] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d7c76b6dcd8945cd818b8107caf4a174)
I20260806 04:08:33.211901 9854 raft_consensus.cc:2781] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:08:33.172106 9771 raft_consensus.cc:482] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.294613 4816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.295269 9771 raft_consensus.cc:504] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.295629 9869 leader_election.cc:288] T 7b1f802648ed4dccb7235c747e9b1553 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.297238 9771 leader_election.cc:288] T a0ff8f19c60b43b99aa42dd6c9cee458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.297478 9795 leader_election.cc:288] T 54785970285b4054b83ec528e156c25c P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 11 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.303750 9600 raft_consensus.cc:504] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.238819 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:33.155212 9885 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.171204 8797 consensus_queue.cc:260] T b75ec76cc4cd43bc90264007dff133f3 P 22fe557f852d4415a96fd683f970cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.239940 9763 raft_consensus.cc:482] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.221834 9812 raft_consensus.cc:686] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [term 9 LEADER]: Becoming Leader. State: Replica: 99690415a7b3441fbdb5e943667556a4, State: Running, Role: LEADER
I20260806 04:08:33.181897 9757 leader_election.cc:288] T 0940457704114112b22dab24a18b43b6 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.241101 9685 raft_consensus.cc:482] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 22fe557f852d4415a96fd683f970cc24)
I20260806 04:08:33.322957 9848 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.323129 9854 raft_consensus.cc:686] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [term 10 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
W20260806 04:08:33.287489 9747 consensus_meta.cc:287] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24: Time spent flushing consensus metadata: real 0.597s user 0.011s sys 0.002s
I20260806 04:08:33.323501 9848 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.324199 9747 raft_consensus.cc:504] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.243104 9651 raft_consensus.cc:482] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.326936 9651 raft_consensus.cc:504] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.329200 9651 leader_election.cc:288] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.329885 4919 raft_consensus.cc:1226] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Rejecting Update request from peer d7c76b6dcd8945cd818b8107caf4a174 for earlier term 7. Current term is 8. Ops: []
I20260806 04:08:33.332885 4919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55924 (request call id 564) took 1851 ms. Trace:
I20260806 04:08:33.244032 9900 raft_consensus.cc:482] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.333572 9900 raft_consensus.cc:504] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.146898 9360 raft_consensus.cc:504] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.335954 9900 leader_election.cc:288] T 7b4bc22d4f8c4b178aca679682ba4b64 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.336225 9360 leader_election.cc:288] T 50ac9e0fb0ae490c88fc2d1584718458 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.244102 9879 leader_election.cc:288] T 21b536e5a86f4281a4d9ee87a11d409f P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.244428 9901 raft_consensus.cc:2781] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:33.337497 9901 raft_consensus.cc:482] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:08:33.337684 5038 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34050 (request call id 1854) took 2638 ms (client timeout 1918 ms). Trace:
I20260806 04:08:33.337920 9901 raft_consensus.cc:3037] T 8109ccbed70f46e0932c03eaddc3caf7 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:08:33.338135 5038 rpcz_store.cc:269] 0806 04:08:30.699706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:30.826276 (+126570us) service_pool.cc:227] Handling call
0806 04:08:33.337435 (+2511159us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b5400016b54. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995e2ae 00007f23de89bc27 00007f23d7f83c04 00007f23d7f83b92 00007f23d7f83ac1 00007f23d7f83a3c 00007f23d7f82cf2 00007f23d7d8494c 00007f23d7d83d86 00007f23d7d8372f
0806 04:08:33.337570 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1752971136}
I20260806 04:08:33.244701 9912 raft_consensus.cc:2781] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:08:33.111735 9612 raft_consensus.cc:504] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.190438 5026 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:33.224215 4928 rpcz_store.cc:276] 0806 04:08:31.480727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:33.195180 (+1714453us) service_pool.cc:227] Handling call
0806 04:08:33.195469 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:33.195653 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:08:33.195680 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:08:33.195718 (+ 38us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:08:33.195752 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:33.195805 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:08:33.195948 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:08:33.342334 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f3bf4a49a4d108609efd686fb6c85" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.342867 4928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.248948 4813 raft_consensus.cc:2454] T 2ea51188c553493cbb9b842b0040bd4f P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:33.344151 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf3e28a45b44b0e9c9f9f7ef333c8f5" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.344723 4813 raft_consensus.cc:2454] T 6bf3e28a45b44b0e9c9f9f7ef333c8f5 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 7.
I20260806 04:08:33.345427 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ae3b855e0d4f98a67493c708f71d33" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:33.237799 9882 log.cc:924] Time spent T 78dcfac52d99420a82f5c7b402a782d7 P 99690415a7b3441fbdb5e943667556a4: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 04:08:33.237056 4924 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 22fe557f852d4415a96fd683f970cc24 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 5.
I20260806 04:08:33.348570 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24"
I20260806 04:08:33.349201 5812 leader_election.cc:302] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24, 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:33.283234 4879 leader_election.cc:302] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 22fe557f852d4415a96fd683f970cc24; no voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a
W20260806 04:08:33.350322 4879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 67184.9us
I20260806 04:08:33.246441 9450 leader_election.cc:288] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.237064 9831 raft_consensus.cc:2781] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:08:33.287075 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f573a65ecea34309b58a81b5fcbfdc02" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.339198 9912 raft_consensus.cc:686] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [term 7 LEADER]: Becoming Leader. State: Replica: 22fe557f852d4415a96fd683f970cc24, State: Running, Role: LEADER
I20260806 04:08:33.358222 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.358923 5037 raft_consensus.cc:2454] T 4219680d51af4d0587436328294ba316 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 6.
I20260806 04:08:33.358819 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940457704114112b22dab24a18b43b6" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.287464 9245 leader_election.cc:288] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.359812 4813 raft_consensus.cc:2454] T 0940457704114112b22dab24a18b43b6 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:33.289062 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27545c8363cd404189c9e93f3bd1426c" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:33.289981 5814 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.019s sys 0.028s
W20260806 04:08:33.361135 5814 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.820s user 0.020s sys 0.028s
I20260806 04:08:33.291225 9769 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.358578 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a878f4d22e44a290e162038ddd83e1" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.291498 8812 leader_election.cc:288] T 30e5d6887268432aaf472e51f9ae0444 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.292977 9811 leader_election.cc:288] T 96c2cb602ae44188a8a52e2cecd5a807 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.164007 9749 consensus_queue.cc:237] T cf27da0cc45d4501b4dac0fbb6fedc16 P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.293905 9610 leader_election.cc:288] T cfd13a1778034c599358d37b5d72c374 P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.163290 9443 raft_consensus.cc:504] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.308499 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b75ec76cc4cd43bc90264007dff133f3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.380956 4804 raft_consensus.cc:2379] T b75ec76cc4cd43bc90264007dff133f3 P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:33.309986 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98064c6d64f74659bfa102bd5a341635" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.306740 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997e4d9ff3ca465199a888e8af23ed7b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.162396 9817 leader_election.cc:288] T a80ee0f1cc7747e8b736886e63200f37 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857)
I20260806 04:08:33.318334 5029 raft_consensus.cc:2454] T cf27da0cc45d4501b4dac0fbb6fedc16 P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fe557f852d4415a96fd683f970cc24 in term 5.
I20260806 04:08:33.321801 9812 consensus_queue.cc:237] T 9b0f147300804e5fab336f0242e01ec4 P 99690415a7b3441fbdb5e943667556a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.322669 9723 raft_consensus.cc:3037] T 793f3bf4a49a4d108609efd686fb6c85 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:33.388749 9817 raft_consensus.cc:2781] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:08:33.389155 9817 raft_consensus.cc:482] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:33.389242 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c567d00bdc4e3b800967318816ba4d" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.389518 9817 raft_consensus.cc:3037] T 2e4a7d4c7807476384d5ce509d4029f9 P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:08:33.323290 9685 raft_consensus.cc:504] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.336544 9763 raft_consensus.cc:504] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.392897 9685 leader_election.cc:288] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.371866 9417 raft_consensus.cc:482] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.339382 5038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5854d52dcd124250a917b57b14dd2afc" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.337502 4919 rpcz_store.cc:276] 0806 04:08:31.481035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:08:33.194417 (+1713382us) service_pool.cc:227] Handling call
0806 04:08:33.194795 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:08:33.330281 (+135486us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:08:33.332579 (+ 2298us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540018fbd4. stack: 00007f23d6a00033 00007f23d69fff30 00007f23d635e534 00007f23e006a8b4 00007f23e006a849 00007f23d98df471 00007f23d995af51 00007f23d995999e 00007f23de89b2e9 00007f23d7f81684 00007f23d7f81612 00007f23d7f81541 00007f23d7f814bc 00007f23d7f80772 00007f23d7d8494c 00007f23d7d83d86
0806 04:08:33.332778 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310193408}
I20260806 04:08:33.415896 4804 raft_consensus.cc:2396] T 28c567d00bdc4e3b800967318816ba4d P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 04:08:33.330277 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5666124a588f45948039fcd27abc9cf4" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"
I20260806 04:08:33.417773 5030 raft_consensus.cc:2379] T 5666124a588f45948039fcd27abc9cf4 P 99690415a7b3441fbdb5e943667556a4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 in current term 4: Already voted for candidate 99690415a7b3441fbdb5e943667556a4 in this term.
I20260806 04:08:33.343360 5026 raft_consensus.cc:2396] T 793f3bf4a49a4d108609efd686fb6c85 P 99690415a7b3441fbdb5e943667556a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7c76b6dcd8945cd818b8107caf4a174 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 04:08:33.326428 9747 leader_election.cc:288] T b5088de56cbc4281b755364a8fddd084 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 8 election: Requested vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.347580 4923 raft_consensus.cc:2454] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 7.
I20260806 04:08:33.171087 9857 raft_consensus.cc:504] T 457d26714def4b128d6f433119c45d5f P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.306627 9600 leader_election.cc:288] T 7cb317a04be74515913f7aee753c193b P 99690415a7b3441fbdb5e943667556a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.370811 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7a93f69159461280c8b02b8ff62358" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.367759 5028 raft_consensus.cc:2454] T 77a878f4d22e44a290e162038ddd83e1 P 99690415a7b3441fbdb5e943667556a4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 8.
I20260806 04:08:33.371335 4928 raft_consensus.cc:2454] T 98064c6d64f74659bfa102bd5a341635 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 7.
I20260806 04:08:33.439853 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12fab895c8c34b9d82a95dc9f77980c6" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:33.376981 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659c18c5db84b729be8dce3bcb9dd32" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.379815 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "589278a46368433c87a150021544ba2e" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.382927 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.382310 4809 raft_consensus.cc:2454] T 997e4d9ff3ca465199a888e8af23ed7b P df2f1e7811114b8a995f6771fb6a6c8a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 11.
I20260806 04:08:33.386024 4805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4bb640df6c2435ea743b7157dd130bd" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:33.442883 4805 raft_consensus.cc:3037] T a4bb640df6c2435ea743b7157dd130bd P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:08:33.358224 9627 raft_consensus.cc:2726] T eaa370887c344934b64e0b3f59c6a3b0 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:08:33.391244 9443 leader_election.cc:288] T 3ff5845199524dcb8deb5af416122df3 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.392951 9763 leader_election.cc:288] T fdc0072b6de7424caf2520a26bdcd0b0 P 22fe557f852d4415a96fd683f970cc24 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df2f1e7811114b8a995f6771fb6a6c8a (127.2.248.65:43865), d7c76b6dcd8945cd818b8107caf4a174 (127.2.248.68:38873)
I20260806 04:08:33.250108 4817 raft_consensus.cc:2454] T 654e437ca202414286305ce8a8c5c4c2 P df2f1e7811114b8a995f6771fb6a6c8a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99690415a7b3441fbdb5e943667556a4 in term 6.
I20260806 04:08:33.456847 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c2cb602ae44188a8a52e2cecd5a807" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
W20260806 04:08:33.457033 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 180112us
I20260806 04:08:33.325874 9848 leader_election.cc:288] T 1e1ae23ce1104b3688f3f881c64ed473 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.371718 9831 raft_consensus.cc:686] T 589278a46368433c87a150021544ba2e P df2f1e7811114b8a995f6771fb6a6c8a [term 6 LEADER]: Becoming Leader. State: Replica: df2f1e7811114b8a995f6771fb6a6c8a, State: Running, Role: LEADER
I20260806 04:08:33.464282 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.417650 5812 leader_election.cc:302] T 7659c18c5db84b729be8dce3bcb9dd32 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:33.440668 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dcfac52d99420a82f5c7b402a782d7" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.443213 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fef94f36fa34d64a4485e7136f39f85" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.467839 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:33.468801 4809 raft_consensus.cc:2379] T f8214005d1a744f9b566b933df1cfe4a P df2f1e7811114b8a995f6771fb6a6c8a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 5: Already voted for candidate df2f1e7811114b8a995f6771fb6a6c8a in this term.
I20260806 04:08:33.450582 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2ae1bef38445b908545a06111f237" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.454916 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80ee0f1cc7747e8b736886e63200f37" candidate_uuid: "d7c76b6dcd8945cd818b8107caf4a174" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.387995 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.458257 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4219680d51af4d0587436328294ba316" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:33.470834 5812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.67:43701, user_credentials={real_user=slave}} blocked reactor thread for 53303.5us
I20260806 04:08:33.472841 5812 leader_election.cc:302] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:33.474874 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa361eb5ab5463895c3f7af5e87237c" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.476222 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.477308 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141037797aa84869bb31e99383874122" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
W20260806 04:08:33.476955 5816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.65:43865, user_credentials={real_user=slave}} blocked reactor thread for 348911us
I20260806 04:08:33.437188 9234 raft_consensus.cc:482] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:08:33.437043 9417 raft_consensus.cc:504] T 21b536e5a86f4281a4d9ee87a11d409f P d7c76b6dcd8945cd818b8107caf4a174 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
I20260806 04:08:33.437408 9854 consensus_queue.cc:237] T 228767bbb3b643c9977ce6fc5b074827 P df2f1e7811114b8a995f6771fb6a6c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.391266 9912 consensus_queue.cc:237] T ebcff471d5ff446ba22c34ec3724dfeb P 22fe557f852d4415a96fd683f970cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.556504 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.799875 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8214005d1a744f9b566b933df1cfe4a" candidate_uuid: "22fe557f852d4415a96fd683f970cc24" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.800364 5035 raft_consensus.cc:2454] T f8214005d1a744f9b566b933df1cfe4a P 99690415a7b3441fbdb5e943667556a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 5.
I20260806 04:08:33.540812 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.468251 4924 raft_consensus.cc:2454] T 5666124a588f45948039fcd27abc9cf4 P 22fe557f852d4415a96fd683f970cc24 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76b6dcd8945cd818b8107caf4a174 in term 4.
I20260806 04:08:33.482210 4817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff5845199524dcb8deb5af416122df3" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" is_pre_election: true
I20260806 04:08:33.538874 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb317a04be74515913f7aee753c193b" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.760759 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e5d6887268432aaf472e51f9ae0444" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d7c76b6dcd8945cd818b8107caf4a174" is_pre_election: true
I20260806 04:08:33.566123 4770 leader_election.cc:302] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:33.802170 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ff8f19c60b43b99aa42dd6c9cee458" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.803177 9877 raft_consensus.cc:2781] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:08:33.803581 9877 raft_consensus.cc:482] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:08:33.803814 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8570417a88db412294c21c9108061c30" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.803923 9877 raft_consensus.cc:3037] T a80ee0f1cc7747e8b736886e63200f37 P df2f1e7811114b8a995f6771fb6a6c8a [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:08:33.805243 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d20919f4994e71841cccb9546444a3" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.806311 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa370887c344934b64e0b3f59c6a3b0" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
W20260806 04:08:33.806617 4770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.68:38873, user_credentials={real_user=slave}} blocked reactor thread for 240613us
I20260806 04:08:33.518079 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5088de56cbc4281b755364a8fddd084" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "22fe557f852d4415a96fd683f970cc24" is_pre_election: true
I20260806 04:08:33.538199 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcff471d5ff446ba22c34ec3724dfeb" candidate_uuid: "99690415a7b3441fbdb5e943667556a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df2f1e7811114b8a995f6771fb6a6c8a"
I20260806 04:08:33.758880 5856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59650 (request call id 755) took 2358 ms. Trace:
I20260806 04:08:33.811009 4813 raft_consensus.cc:2379] T ebcff471d5ff446ba22c34ec3724dfeb P df2f1e7811114b8a995f6771fb6a6c8a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 99690415a7b3441fbdb5e943667556a4 in current term 7: Already voted for candidate 22fe557f852d4415a96fd683f970cc24 in this term.
I20260806 04:08:33.518898 5812 leader_election.cc:302] T 77a878f4d22e44a290e162038ddd83e1 P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174; no voters:
I20260806 04:08:33.518321 9848 raft_consensus.cc:2781] T 4219680d51af4d0587436328294ba316 P d7c76b6dcd8945cd818b8107caf4a174 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:08:33.812682 9685 raft_consensus.cc:482] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 99690415a7b3441fbdb5e943667556a4)
I20260806 04:08:33.488766 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086f8af9433f44aa8aa4fccbfa90eb02" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.812669 5816 leader_election.cc:302] T 997e4d9ff3ca465199a888e8af23ed7b P d7c76b6dcd8945cd818b8107caf4a174 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 99690415a7b3441fbdb5e943667556a4, d7c76b6dcd8945cd818b8107caf4a174, df2f1e7811114b8a995f6771fb6a6c8a; no voters:
I20260806 04:08:33.813716 9685 raft_consensus.cc:504] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" member_type: VOTER last_known_addr { host: "127.2.248.65" port: 43865 } } peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } }
I20260806 04:08:33.814169 5026 raft_consensus.cc:2454] T 086f8af9433f44aa8aa4fccbfa90eb02 P 99690415a7b3441fbdb5e943667556a4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2f1e7811114b8a995f6771fb6a6c8a in term 6.
I20260806 04:08:33.814224 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f147300804e5fab336f0242e01ec4" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4" is_pre_election: true
I20260806 04:08:33.815922 9685 leader_election.cc:288] T f573a65ecea34309b58a81b5fcbfdc02 P df2f1e7811114b8a995f6771fb6a6c8a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 22fe557f852d4415a96fd683f970cc24 (127.2.248.66:34857), 99690415a7b3441fbdb5e943667556a4 (127.2.248.67:43701)
I20260806 04:08:33.517740 9234 raft_consensus.cc:504] T 1e1ae23ce1104b3688f3f881c64ed473 P 22fe557f852d4415a96fd683f970cc24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22fe557f852d4415a96fd683f970cc24" member_type: VOTER last_known_addr { host: "127.2.248.66" port: 34857 } } peers { permanent_uuid: "99690415a7b3441fbdb5e943667556a4" member_type: VOTER last_known_addr { host: "127.2.248.67" port: 43701 } } peers { permanent_uuid: "d7c76b6dcd8945cd818b8107caf4a174" member_type: VOTER last_known_addr { host: "127.2.248.68" port: 38873 } attrs { promote: false } }
W20260806 04:08:33.817324 4768 connection.cc:232] RPC call timeout handler was delayed by -1.3957051229663193s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260806 04:08:33.816150 5026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1f802648ed4dccb7235c747e9b1553" candidate_uuid: "df2f1e7811114b8a995f6771fb6a6c8a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "99690415a7b3441fbdb5e943667556a4"